Java SDK fat JAR native bindings in Kubernetes

I'm encountering a discrepancy in Docker vs. Kubernetes when running my Spring Boot application, which uses the ziti-sdk-jvm SDK. On start, my application successfully creates a Ziti context using an identity pulled from a secret and the controller logs the services that are available. The application then creates a OkHttpClient client using the Ziti socket factory and DNS resolver (like the SDK samples):

@Bean
public OkHttpClient zitiClient() throws Exception {
  final KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
  final TrustManagerFactory tmf =
      TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());

  tmf.init(ks);

  final X509TrustManager tm = (X509TrustManager) tmf.getTrustManagers()[0];

  return new OkHttpClient.Builder().socketFactory(Ziti.getSocketFactory())
      .sslSocketFactory(Ziti.getSSLSocketFactory(), tm).dns(hostname -> {
        InetAddress address = Ziti.getDNSResolver().resolve(hostname);
        if (address == null) {
          address = InetAddress.getByName(hostname);
        }
        return address != null ? Collections.singletonList(address)
            : Collections.emptyList();
      }).callTimeout(5, TimeUnit.MINUTES).build();
}

This client is then utilized to send REST requests to the available Ziti services. In Docker, where we build and run the application using Maven, this works as expected. In Kubernetes, where we build using Maven and deploy an image with the built JAR, we get the following exception:

Exception in thread "DefaultDispatcher-worker-2" java.lang.NoClassDefFoundError: Could not initialize class org.openziti.crypto.Crypto
    at org.openziti.net.ZitiSocketChannel.connectInternal$ziti(ZitiSocketChannel.kt:149)
    at org.openziti.net.ZitiSocketChannel$connectInternal$1.invokeSuspend(ZitiSocketChannel.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
    Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@596adab6, Dispatchers.IO]
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.IllegalArgumentException: URI is not hierarchical [in thread "DefaultDispatcher-worker-1"]
    at java.base/java.io.File.<init>(File.java:420)
    at com.goterl.resourceloader.ResourceLoader.urlToFile(ResourceLoader.java:588)
    at com.goterl.resourceloader.ResourceLoader.urlToFile(ResourceLoader.java:567)
    at com.goterl.resourceloader.ResourceLoader.extractFilesOrFoldersFromJar(ResourceLoader.java:227)
    at com.goterl.resourceloader.ResourceLoader.nestedExtract(ResourceLoader.java:174)
    at com.goterl.resourceloader.ResourceLoader.extractFromWithinAJarFile(ResourceLoader.java:98)
    at com.goterl.resourceloader.ResourceLoader.copyToTempDirectory(ResourceLoader.java:80)
    at com.goterl.resourceloader.SharedLibraryLoader.load(SharedLibraryLoader.java:53)
    at com.goterl.lazysodium.utils.LibraryLoader.loadBundledLibrary(LibraryLoader.java:134)
    at com.goterl.lazysodium.utils.LibraryLoader.loadLibrary(LibraryLoader.java:107)
    at com.goterl.lazysodium.SodiumJava.<init>(SodiumJava.java:34)
    at org.openziti.crypto.JavaCryptoLoader.load(JavaCryptoLoader.kt:26)
    at org.openziti.crypto.Crypto.<clinit>(Crypto.kt:49)
    ... 10 more
2025-06-04T14:52:56.466Z ERROR 6 --- [atcher-worker-2] ziti-conn[lawgbpxdp/2]                   :  failed to connect: java.lang.NoClassDefFoundError: Could not initialize class org.openziti.crypto.Crypto
2025-06-04T14:53:06.359Z  WARN 6 --- [health:8765/...] o.o.net.ZitiSocketFactory$ZitiConnector  : lawgbpxdp: java.net.SocketTimeoutException
2025-06-04T14:53:06.359Z  INFO 6 --- [health:8765/...] o.o.net.ZitiSocketFactory$ZitiConnector  : no ZitiContext provides service for <service_name>/<ip_address>:<port>
2025-06-04T14:53:16.363Z ERROR 6 --- [nio-8080-exec-7] c.r.e.e.ControllerExceptionHandler       : Internal Server Error

java.lang.RuntimeException: Ziti request failed: timeout

After doing some digging, this seems to suggest that the Ziti SDK is failing to load native dependencies that are required by org.openziti.crypto.Crypto. Specifically "URI is not hierarchical" suggests that this doesn't work in Kubernetes because the resource loader is trying to extract the libraries using a nested jar:file: URL from the deployed fat JAR.

Any assistance would be appreciated, thanks.

hi @sdundas, that sure seems strange. It definitely appears related to the libsodium library. Unfortunately, I found this bug that someone from NetFoundry (the OpenZiti sponsor) opened over a year ago...

I'm not sure if we're going to have a great answer here. I'll have to bring it up with the broader team and see if we have any options or workarounds to share...

Hi @sdundas, this is actually a bug with the resource loader that Spring Boot uses. The bug for that is here: Spring Boot 3.2 nested jar references not supported ยท Issue #18 ยท terl/resource-loader ยท GitHub. Our patch has been merged but not release yet.

Until the fix is released the workaround we're using internally is to unpack libsodium when the app is launched. This can be done via Spring Boot, and configured by the mvn or gradle spring boot plugins.

Gradle:

bootJar {
    requiresUnpack '**/lazysodium-java-*.jar'
}

Maven

<plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <configuration>
          <requiresUnpack>
              <dependency>
                  <groupId>com.goterl</groupId>
                  <artifactId>lazysodium-java</artifactId>
              </dependency>
          </requiresUnpack>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>build-info</goal>
                <goal>repackage</goal>
            </goals>
          </execution>
        </executions>
</plugin>
2 Likes