Both the controller and the tunnelers are running on Ubuntu 22.04 so I will wait for the fix before upgrading.
Thanks for the heads up!
Can you describe “performance”? Is it throughput, session creation, something else?
You say you only have one Edge Router, but the services seem to indicate multiple services in Europe. Are those just names you set up, or are they actually in Europe? And where are the accessing endpoints?
Not a throughput issue, It looks more as unresponsive. I just tested it right now and this is what I captured on my laptop.
You can see a 10-second delay before receiving a response from the server.
That 10-second delay it is what I got right now, sometimes it much worse and can take a few minutes.
And here is the same pcap but with traffic going to one of my internal applications, which is hosted in another tunneler:
I can confirm that the applications are not the issue, they work perfectly fine from the VPN or from the offices, slowness is only with OpenZiti
There are multiple services which are "hosted" in multiples tunnelers. I have only 1 OpenZiti controller and 1 OpenZiti router.
I am located in Europe and all these services and tunnelers are hosted in AWS Ireland
OK, that looks a lot like there is call setup trouble. In the controller and router logs, you should find messages that would indicate if there are retries. Below is one such message. You can probably grep "received failed route status" in the controller log. As you can see in the entry, the retry to route a failed attempt is 5 seconds. I would guess you are succeeding on a later attempt. The "r/XXXXX" value is the router ID you can get from the CLI "ziti fabric list routers". Since you only have one, that is meaningless, but wanted to point it out if you grow to more routers. If you are logging them curently, the circuit logs will include the successes, and you can get the terminator ID from those records. It might be interesting to see what terminators are successfully being used vs. not, if there is a pattern, so you can examine the tunneler and see if there is an issue at that end.
Aug 17 18:08:41 ip-10-19-2-255 ziti-controller: {"apiSessionId":"clXXXXXXXXXXX","channels":["establishPath"],"circuitId":"hkXXXXXXX","file":"github.com/openziti/fabric@v0.17.2/controller/network/routesender.go:115","func":"github.com/openziti/fabric/controller/network.(*routeSender).route","level":"warning","msg":"received failed route status from [r/XXXXXXXXX] for attempt [#2] of [s/hkXXXXXXXXX] (error creating route for [c/hkXXXXXXX]: timeout waiting for response after 5s)","sessionId":"clXXXXXXXX","time":"2023-08-17T18:08:41.145Z"}
2 Likes
I just did this. In the past 5 days I have 400k messages like this:
Aug 13 06:06:27 openziti-controller ziti-controller[484]: {"_channels":["selectPath"],"apiSessionId":"cll8y165fx7stdglm60rpllm7","attemptNumber":2,"circuitId":"GsHiryDCA","error":"error creating route for [s/GsHiryDCA] on [r/jftypF6Xp-] (error creating route for [c/GsHiryDCA]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/fabric@v0.22.7/controller/network/network.go:490","func":"github.com/openziti/fabric/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","serviceId":"2GuHHNHwTZiZE0mAW3Osdt","serviceName":"grafana-nl.service","sessionId":"cll8yp4n4x8obdglmtes7vev2","time":"2023-08-13T06:06:27.023Z"}
Is this because of a terminator failing? From those logs, is it possible to find out which one it is?
that sounds like invalid terminators are in play. these should be alleviated once the new version is usable
1 Like
Hey, is it possible to get notified once this happens?
Thanks!!
Hi, sorry I never closed the loop with you here. The updated build with openssl libraries was 0.22.3, but a lot has happened since then. Specifically we realized that some people are running controllers with ca bundles that the openssl libraries don't use for validation. So in the meantime we have gone back to using mbedtls, which is linked statically into the ziti-edge-tunnel binary.
and for the record the version with mbedtls is 0.22.7.
ahh great!
So it should be safe for me to upgrade to version 0.22.7 with the controller running in Ubuntu 22.04, right?
Yes, we have all of the shared library issues sorted out now, and the ca cert issues are addressed for now by going back to mbedtls.
1 Like