ZET does not recover after prolonged controller/network outage once API session expires

Tunneler Apps Ziti Edge Tunnel
1

I am seeing ziti-edge-tunnel fail to recover after controller and network connectivity are restored following a prolonged outage.

A short network interruption appears to be handled normally. The problem occurs when connectivity remains unavailable long enough for the existing API/OIDC session to expire.

Environment

  • ziti-edge-tunnel: using Ziti C SDK 1.18.4
  • Ziti Tunneler SDK: 1.18.5
  • Controller: v2.0.1
  • Edge Routers: v2.0.1
  • Authentication method: OIDC
  • Linux/systemd

The tunneler has one identity and normally connects successfully to the controller and two edge routers.

Observed behavior

At startup, the tunneler authenticates normally:

connected to controller https://<controller>:443 version v2.0.1
using OIDC authentication method
authorization successful
context event : status is OK
router <edge-router-1> connected
router <edge-router-2> connected

At approximately 18:00:48, network/controller connectivity was lost. Both edge-router channels disconnected and the controller became unavailable:

channel disconnected [-103/software caused connection abort]
disconnected from edge router[<edge-router-1>]
reconnecting in 5600ms

channel disconnected [-103/software caused connection abort]
disconnected from edge router[<edge-router-2>]

request[/services/.../edge-routers] failed
no controllers are online

The tunneler continued retrying, which is the behavior I would expect.

After the outage had lasted approximately 27 minutes, the authentication session expired:

18:27:33 WARN ziti-sdk:ziti.c:236 ztx_set_unauthenticated()
ztx[1] auth error: session token has expired

WARN tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event()
ziti_ctx controller connections failed: not authorized

Shortly afterward, the SDK detected that there was no longer an API session and attempted to re-authenticate:

WARN ziti-sdk:ziti_ctrl.c:750 verify_api_session()
ctrl[https://<controller>:443] no API session

ERROR ziti-sdk:ziti.c:1756 update_identity_data()
ztx[1] failed to get identity_data:
no api session token set for ziti_controller[UNAUTHORIZED]

WARN ziti-sdk:ziti.c:1758 update_identity_data()
ztx[1] api session is no longer valid. Trying to re-auth

WARN ziti-sdk:ziti.c:236 ztx_set_unauthenticated()
ztx[1] auth error: no api session token set for ziti_controller

OIDC refresh attempts also continued failing:

WARN ziti-sdk:oidc.c:726 oidc_refresh_cb()
oidc[internal] OIDC token refresh failed (...), attempt 5

The edge-router connection logic then delayed reconnecting because the Ziti context was no longer authenticated:

INFO ziti-sdk:channel.c:934 reconnect_cb()
ch[0] ztx[1] is not fully authenticated (auth_state[0]), delaying re-connect

Failure to recover

After network and controller connectivity were restored, the existing ziti-edge-tunnel process did not recover.

It remained in the same state for many hours:

no API session
no api session token set for ziti_controller
api session is no longer valid. Trying to re-auth

The OIDC refresh counter eventually reached more than 6,800 attempts. Near the end I also observed:

OIDC token refresh failed (-103/software caused connection abort), attempt 6871
OIDC token refresh failed (-9/bad file descriptor), attempt 6872
OIDC token refresh failed (-9/bad file descriptor), attempt 6873

The process itself remained running, so systemd did not automatically restart it.

Restart immediately restores connectivity

When ziti-edge-tunnel was restarted, it recovered immediately.

The new process:

  1. Initialized the controller.
  2. Initialized the OIDC provider.
  3. Requested a token.
  4. Reported authorization successful.
  5. Connected to the controller.
  6. Reloaded the services.
  7. Connected to both edge routers.

For example:

connected to controller https://<controller>:443 version v2.0.1
using OIDC authentication method
authorization successful
ziti_ctx[...] connected to controller
context event : status is OK

router <edge-router-1> connected
router <edge-router-2> connected

Normal hosted-service connections then resumed.

This suggests that underlying network connectivity, controller availability, identity configuration, and router/service policies were valid once connectivity was restored. Reinitializing the tunneler/OIDC state by restarting the process was sufficient to recover.

Expected behavior

I would expect ziti-edge-tunnel to recover automatically when:

  1. Controller/network connectivity is lost.
  2. The API/OIDC session expires during the outage.
  3. Network connectivity is later restored.

If the previous API session can no longer be refreshed, I would expect the SDK to establish a new authenticated session rather than remain indefinitely in the unauthenticated state.

Possible reproduction

I believe this may be reproducible by:

  1. Start ziti-edge-tunnel and verify successful controller and edge-router connections.
  2. Block network access to the controller and edge routers.
  3. Leave connectivity blocked long enough for the existing authentication/API session to expire.
  4. Restore network access.
  5. Observe whether the tunneler creates a new authenticated API session and reconnects to the edge routers.

In my test, the important transition occurred after roughly 27 minutes of outage, when the log changed from CONTROLLER_UNAVAILABLE to:

session token has expired
not authorized
no API session
Trying to re-auth

After that transition, the process never recovered on its own.