Invalid session for only zac, other services work

1

I’ve set up OpenZiti like so: How to for self hosting behind NAT? - #20 by ZerotrustExplorer

After working for a bit in the ZAC to figure out how to add services, I setup a service for a vm running nginx in default mode, then jellyfin, both working perfectly.

Some time later I lost connection to the ZAC, unreachable from the client running the ziti tunneler on windows.
nginx and jellyfin are accessible.

windows client, service log is showing:

[2025-10-30T15:02:48.537Z]   ERROR ziti-sdk:connect.c:1068 connect_reply_cb() conn[2.350/6so7MLlF/Connecting](secure-apis) failed to connect, reason=invalid session
[2025-10-30T15:02:48.537Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed

bit of fast scrolling edge router container:

[ 535.814]    INFO ziti/router/xgress_edge.(*hostedServiceRegistry).Remove: {reason=[invalid session] terminatorId=[5ImZ7TnJciGyHGfu9BktOM]} terminator removed from router set
[ 538.246]    INFO ziti/router/xgress_edge.(*edgeClientConn).processBindV2 [ch{edge}->u{classic}->i{ziti-sdk-c[0]@zititest/PB36}]: {terminatorId=[3RMILZLKwgO4sLjg2YkMkx] connId=[1] type=[EdgeBindType] apiSessionToken=[map[apiSessionId:cmhciom3y000r0b9g1kbwhewb certFingerprints:[e286670ecb8496d2e1dde2d61598e384e70c9941] identityId:eBQaOXuEDm tokenId:gR2u8Q465CKiUwlScrWi1Jno1FA type:legacyProtobuf]] listenerId=[2G=��ˤN�2�����q����������c���] routerId=[.B7V21VKp8] edgeSeq=[0] serviceSessionToken=[map[apiSessionId:cmhciom3y000r0b9g1kbwhewb identityId:eBQaOXuEDm serviceId:43goV7ylfrfVLQAsiuGvko tokenId:cmhdkex2n00kd0b9g0def68j1 type:JWT]] chSeq=[228] bindConnId=[1]} establishing terminator
[ 538.246]    INFO ziti/router/xgress_edge.(*hostedServiceRegistry).evaluateEstablishQueue: {serviceSessionTokenId=[cmhdkex2n00kd0b9g0def68j1] terminatorId=[3RMILZLKwgO4sLjg2YkMkx] state=[establishing]} queuing terminator to send create
[ 538.246]    INFO ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminator: {apiSessionToken=[map[apiSessionId:cmhciom3y000r0b9g1kbwhewb certFingerprints:[e286670ecb8496d2e1dde2d61598e384e70c9941] identityId:eBQaOXuEDm tokenId:gR2u8Q465CKiUwlScrWi1Jno1FA type:legacyProtobuf]] terminatorId=[3RMILZLKwgO4sLjg2YkMkx] serviceSessionToken=[map[apiSessionId:cmhciom3y000r0b9g1kbwhewb identityId:eBQaOXuEDm serviceId:43goV7ylfrfVLQAsiuGvko tokenId:cmhdkex2n00kd0b9g0def68j1 type:JWT]] routerId=[.B7V21VKp8]} sending create terminator v2 request
[ 538.248]    INFO ziti/router/xgress_edge.(*hostedServiceRegistry).Remove: {terminatorId=[3RMILZLKwgO4sLjg2YkMkx] reason=[invalid session]} terminator removed from router set

controller:

[ 563.060]   ERROR ziti/controller/handler_edge_ctrl.(*baseSessionRequestContext).loadFromBolt: {operation=[create.terminator] error=[invalid session]} invalid session
[ 563.061]   ERROR ziti/controller/handler_edge_ctrl.(*createTerminatorV2Handler).returnError [ch{.B7V21VKp8}->u{classic}->i{.B7V21VKp8/bm5b}]: {terminatorId=[798gHpwQQNLN99En4q9PnR] error=[invalid session] routerId=[.B7V21VKp8]} responded with error
[ 563.150]   ERROR ziti/controller/handler_edge_ctrl.(*baseSessionRequestContext).loadFromBolt: {operation=[create.terminator] error=[invalid session]} invalid session
[ 563.150]   ERROR ziti/controller/handler_edge_ctrl.(*createTerminatorV2Handler).returnError [ch{.B7V21VKp8}->u{classic}->i{.B7V21VKp8/bm5b}]: {terminatorId=[3ee47m7pxOeNDdd0JN0JpL] error=[invalid session] routerId=[.B7V21VKp8]} responded with error

Things tried:

  • restarting the ziti tunneler on windows client
  • restarting the OpenZiti containers and VM running the OpenZiti edge router and controller containers

Appears resolved after:

  • disable my identity in Windows OpenZiti client
  • stop service by ‘Tap to disconnect’ in OpenZiti Client
  • press ‘Tap to Connect’, then enabling the identity.
2

Edit: the issue came back, I’ve been restarting the containers and the client but no joy, the other services keep working, weird issue!

Edit: Sometimes after disabling identity off and on it’s happy.

Let me know if I can DM trace logs or something.

3

Hi @ZerotrustExplorer, thanks for providing a bunch of info so far. It's exceptionally odd for other services to work, yah. Is it possible that the Jellyfin client is erroneously caching the ip address of the server? The way our intercept-based tunneling works means that it's possible to get a different IP now and then.

On the whole, would you be willing to clear your logs, set them to trace, replicate the problem and then send me a feedback zip file via DM here on discourse for us to look at?

Is there any other information that is possibly relevant such as, do you possibly use the exact same address via ziti to get to the jellyfin server and that same address is also availalbe on the local network? While that sort of name shadowing generally works fine, I suppose it could possibly be a problem.

4

After a router reboot (it became unresponsive, monitoring showed it used up to 80% of memory) I have not been able to reproduce the issue.

At first I thought it had to do with multiple roles assigned to an identity but that did not reproduce the issue.

If it comes up again I will set debug level to trace and forward some logs.

Couple questions:

  • how should I handle DNS locally and externally? At LAN level, do I point openziti.domain.tld to my WAN address or the VM running the containers itself?
    Currently internal and external DNS points it to my WAN address, which seems to work fine
  • Is it possible to reset an identity for re-enrollment by the ziti command, as one can do by zac?
5

Since you're hosting your whole overlay on your LAN I think this is probably 'the best' way to do it.

I don't believe it's enabled at this time. I'm sure we'll implement it at some point, it's just never been implemented yet.

6

I think I managed to replicate by ‘tap to disconnect’, ‘tap to connect’ on windows client, sending client log by DM.

I do an attempt to reach zac (unsuccessful) then an attempt to reach jellyfin (successful)

7

As a workaround I’ve added this line in my docker-compose.yml

   **`-`** `${ZITI_INTERFACE:-0.0.0.0}:${ZITI_CTRL_SECURE_PORT:-9000}:${ZITI_CTRL_SECURE_PORT:-9000}`

under the ports section of

ziti-controller:

This makes zac available on http://IPOFDOCKERHOST:9000/zac