Java SDK - client didn't send dst_protocol in app_

SDK Questions
1

I have an echo service, and it works fine when I connect via Windows Desktop Edge.
But I'm struggling to connect from a client using the Java SDK.

This code extract fails on the first line

        try (ZitiConnection conn = zitiContext.dial(serviceName)) {
            // Communicate with the service
            try {
                doCommunication(conn);
            } catch (Exception e) {
                throw new Exception("Communication failed", e);
            }
        } catch (Exception e) {
            throw new Exception("Failed to open connection with service", e);
        }

Exception details

Caused by: java.net.ConnectException: exceeded maximum [2] retries creating circuit [c/32Tddt2CK]: error creating route for [s/32Tddt2CK] on [r/SiDCyFNjt] (error creating route for [c/32Tddt2CK]: failed to establish connection with terminator address 3ML6pE8lG401Ne3KLH82CB. error: (rejected by application))
	at org.openziti.net.ZitiSocketChannel.doZitiHandshake$ziti(ZitiSocketChannel.kt:488) ~[ziti-0.27.5.jar:0.27.5]
	at org.openziti.net.ZitiSocketChannel$doZitiHandshake$1.invokeSuspend(ZitiSocketChannel.kt) ~[ziti-0.27.5.jar:0.27.5]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-2.0.20.jar:2.0.20-release-360]
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101) ~[kotlinx-coroutines-core-jvm-1.9.0.jar:?]
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113) ~[kotlinx-coroutines-core-jvm-1.9.0.jar:?]
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89) ~[kotlinx-coroutines-core-jvm-1.9.0.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:589) ~[kotlinx-coroutines-core-jvm-1.9.0.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:823) ~[kotlinx-coroutines-core-jvm-1.9.0.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:720) ~[kotlinx-coroutines-core-jvm-1.9.0.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:707) ~[kotlinx-coroutines-core-jvm-1.9.0.jar:?]

and the service logs contains

[2025-02-28T13:28:07.218Z]   ERROR tunnel-cbs:ziti_hosting.c:602 on_hosted_client_connect() hosted_service[echo.duncan] client[Duncan Client] failed to compute destination protocol: config specifies 'forwardProtocol', but client didn't send dst_protocol in app_
[2025-02-28T13:28:07.236Z]   ERROR tunnel-cbs:ziti_hosting.c:602 on_hosted_client_connect() hosted_service[echo.duncan] client[Duncan Client] failed to compute destination protocol: config specifies 'forwardProtocol', but client didn't send dst_protocol in app_

The service was created in ZAC using Create Simple Service, and it works fine when tunneling using Windows Desktop Edge.

The error seems reasonable, as in Java I'm not specifying which protocol to use (TCP/UDP).
But how to set it?

There is another dial method that takes a ZitiAddress.Dial, but there is no documentation or samples, and I can't figure out how to use the parameters.

        ZitiAddress.Dial dial = new ZitiAddress.Dial(serviceName, appData, "dialIdentity", "dialCallerId");
        try (ZitiConnection conn = zitiContext.dial(dial)) {

Param 1 serviceName - yup, that's straightforward
Param 2 appData - It's just an Object, absolutely no idea whether this could be used to provide the protocol.
Param 3 identity - Might be the terminatorId if there is more than one terminator, but likely to be something else
Param 4 - callerId - Might be setting the the externalId for the application, but could be something else

In summary - when dialing using the Java SDK, how do I set the protocol to be forwarded to the service?

2

Hi @duncan.simey

in order to pass forwarding information you need to use intercept address (otherwise there is nothing to forward)

there is currently two ways to do it:

  1. get a standard Java Socket:
 Socket conn = zitiContext.connect("myhost.ziti", 8080);
  1. get Java nio2 channel:
AsynchronousSocketChannel ch = zitiContext.open();
ch.connect(InetSocketAddress.createUnresolved("myhost.ziti", 8080)).get(timeout, TimeUnit.SECOND);
3

Thanks @ekoby !
Ooooo - that code looks like it might be creating a DNS entry that can be used from outside my application. Might be really useful to me in the future, but I don't think it's what I need right now.

My application is communicating with a service via OpenZiti, there is no other messaging involved. The snippet I included creates a ZitiConnection and I was planning to read() and write() via the connection.
It's not a standard Java socket implementation which is going to cause some compatibility issues for me, but your code snippets give me hope that I may be able to find a solution.

I did manage to get a bit further by looking at the SDK source in Git and found the appData Object is supposed to be a DialData instance.
This snippet appears to be working

        DialData dialData = new DialData(Protocol.TCP, null, null, null, null, null, null, null);
        ZitiAddress.Dial dial = new ZitiAddress.Dial(serviceName, dialData, null, "dialCallerId");
        try (ZitiConnection conn = zitiContext.dial(dial)) {

My next jobs are

  • Figure how the heck to read the ZitiConnection as there is no Java InputStream. I can use the read() method but I need to crowbar it into existing code which is stream based. write() is fine for my purposes.
  • See if I can get my head around your code snippets.
    -- Standard Java Socket, woot woot.
    -- Not sure your snippet does what I need, but I need to confirm this.
    -- If the snippet does what I think it does, then it will be very useful in my future work

The lack of JavaDoc and samples is driving me crazy, but once I get it working I probably won't have to go near it very often as I can focus on capabilities rather than messaging.
I'd be happy to submit my test application for inclusion into the Git samples

4

One note: the DNS entry is only available inside the Java application.

I believe it would be easy to make ZitiConnection to confirm to InputStream and OutputStream (or add them to the interface)

5

Ah.... Inside the App, got it.
Yes, wrapping ZitiConnection with Java IO makes sense. You are right, it doesn't look too hard. I'll give that a go.

6

@ekoby I tried your Socket pattern and it works nicely, except I found a bug in AsynchSocketImpl, probably caused by a problem in InputChannel
What is the proper way to report OpenZiti bugs?

Here's my code

    public static void hitZitiService(@Nonnull ZitiContext zitiContext, @Nonnull String serviceName) throws Exception {
        try (Socket socket = zitiContext.connect(serviceName, 8111)) {
            socket.setSoTimeout(1000);
            InputStream zitiInputStream = socket.getInputStream();

            try {
                byte[] buff = new byte[1024];
                while (true){
                    try {
                        int rxLength = zitiInputStream.read(buff, 0, buff.length);
                        if (rxLength < 0) {
                            logger.info("=== [Received message length = "+rxLength+"]");
                        } else {
                            // Log the message
                            String message = new String(buff, 0, rxLength, StandardCharsets.UTF_8);
                            logger.info("=== " + StringUtils.toPrintable(message, 200));
                        }
                    } catch (SocketTimeoutException e) {
                        logger.debug("Socket timeout, trying again", e);
                    }
                }
            } catch (Exception e) {
                throw new Exception("Failed to receive message", e);
            }
        }
    }

The service sends a welcome message, which is read successfully by the first call to read().
The second call to read() throws a SocketTimeoutException, which is expected as there are no more messages from the service
Then the third call to read() is blocked forever, it should timeout.
It seems one of the mutex locks is not being released when the timeout exception is thrown.

Debugging shows that the second call to read() leaves AsynchSocketImpl:159 rf as [Not completed].

                    val rf = CompletableFuture<Int>()
                    val to = (getOption(SocketOptions.SO_TIMEOUT) as Number).toLong()
                    channel.read(input, 0, TimeUnit.MILLISECONDS, rf,
                        object : CompletionHandler<Int, CompletableFuture<Int>> {

Can't quite get my head around the code, but I think the cause is InputChannel:122 which only completes the handler if data is copied, and timeout exceptions don't copy any data.

        if (copied > 0) {
            inputSupport.mut.unlock()
            handler.completed(copied, att)
            return
        }

FYI - I regularly use socket timeouts to allow applications to shutdown in an orderly manner rather than hang waiting for input they are never going to receive.

Please can you let me know where I should submit this bug report.

FYI - there's another bug I haven't pinned down yet that is not Java specific. My Ziti terminators sometimes disappear leaving the services unusable. Restarting the Ziti tunneller recreates the terminators, but I won't be able to do this for endpoints deployed to hosts where I have no external access. When I work out how to reproduce this reliably, I'll report it.

7

You can create an issue in the GitHub GitHub - openziti/ziti-sdk-jvm: Ziti SDK for JVM. Or I will do it based on discourse posts

Thank you for testing it out.

8

@ekoby Thanks - I'll do it tomorrow

9

What versions of controller/routers/ziti-edge-tunnel are you using?

10

@ekoby I saw most of the problems using two instances of Windows Desktop edge (dial and bind), but I have also seen this on a service tunneled through the Ubuntu public edge router (hosted AWS) which was serving SSH.

Windows desktop edge - ZAC says 1.3.6, application dialog shows 1.3.8, it did update at some point
Ubuntu Ziti edge router - ZAC shows v1.3.3 in the JSON

It's something to do with managing policies. I transitioned from endpoint specific policies to roles, but I think I also saw the problem during initial testing so maybe also policy creation.
When I leave my policies alone it seems stable.

It's not just me. I'm sharing my prototyping experience with another team and it seems they have seen something like this too, very similar symptoms but they hadn't got as far as spotting the missing terminators. They were blaming their Debian edge router implementation, but I think we have the same problem.

11

Bug report submitted to GitHub

1 Like
12

Awesome work - thanks!