I am seeing ziti-edge-tunnel fail to recover after controller and network connectivity are restored following a prolonged outage.
A short network interruption appears to be handled normally. The problem occurs when connectivity remains unavailable long enough for the existing API/OIDC session to expire.
Environment
ziti-edge-tunnel: using Ziti C SDK1.18.4- Ziti Tunneler SDK:
1.18.5 - Controller:
v2.0.1 - Edge Routers:
v2.0.1 - Authentication method: OIDC
- Linux/systemd
The tunneler has one identity and normally connects successfully to the controller and two edge routers.
Observed behavior
At startup, the tunneler authenticates normally:
connected to controller https://<controller>:443 version v2.0.1
using OIDC authentication method
authorization successful
context event : status is OK
router <edge-router-1> connected
router <edge-router-2> connected
At approximately 18:00:48, network/controller connectivity was lost. Both edge-router channels disconnected and the controller became unavailable:
channel disconnected [-103/software caused connection abort]
disconnected from edge router[<edge-router-1>]
reconnecting in 5600ms
channel disconnected [-103/software caused connection abort]
disconnected from edge router[<edge-router-2>]
request[/services/.../edge-routers] failed
no controllers are online
The tunneler continued retrying, which is the behavior I would expect.
After the outage had lasted approximately 27 minutes, the authentication session expired:
18:27:33 WARN ziti-sdk:ziti.c:236 ztx_set_unauthenticated()
ztx[1] auth error: session token has expired
WARN tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event()
ziti_ctx controller connections failed: not authorized
Shortly afterward, the SDK detected that there was no longer an API session and attempted to re-authenticate:
WARN ziti-sdk:ziti_ctrl.c:750 verify_api_session()
ctrl[https://<controller>:443] no API session
ERROR ziti-sdk:ziti.c:1756 update_identity_data()
ztx[1] failed to get identity_data:
no api session token set for ziti_controller[UNAUTHORIZED]
WARN ziti-sdk:ziti.c:1758 update_identity_data()
ztx[1] api session is no longer valid. Trying to re-auth
WARN ziti-sdk:ziti.c:236 ztx_set_unauthenticated()
ztx[1] auth error: no api session token set for ziti_controller
OIDC refresh attempts also continued failing:
WARN ziti-sdk:oidc.c:726 oidc_refresh_cb()
oidc[internal] OIDC token refresh failed (...), attempt 5
The edge-router connection logic then delayed reconnecting because the Ziti context was no longer authenticated:
INFO ziti-sdk:channel.c:934 reconnect_cb()
ch[0] ztx[1] is not fully authenticated (auth_state[0]), delaying re-connect
Failure to recover
After network and controller connectivity were restored, the existing ziti-edge-tunnel process did not recover.
It remained in the same state for many hours:
no API session
no api session token set for ziti_controller
api session is no longer valid. Trying to re-auth
The OIDC refresh counter eventually reached more than 6,800 attempts. Near the end I also observed:
OIDC token refresh failed (-103/software caused connection abort), attempt 6871
OIDC token refresh failed (-9/bad file descriptor), attempt 6872
OIDC token refresh failed (-9/bad file descriptor), attempt 6873
The process itself remained running, so systemd did not automatically restart it.
Restart immediately restores connectivity
When ziti-edge-tunnel was restarted, it recovered immediately.
The new process:
- Initialized the controller.
- Initialized the OIDC provider.
- Requested a token.
- Reported
authorization successful. - Connected to the controller.
- Reloaded the services.
- Connected to both edge routers.
For example:
connected to controller https://<controller>:443 version v2.0.1
using OIDC authentication method
authorization successful
ziti_ctx[...] connected to controller
context event : status is OK
router <edge-router-1> connected
router <edge-router-2> connected
Normal hosted-service connections then resumed.
This suggests that underlying network connectivity, controller availability, identity configuration, and router/service policies were valid once connectivity was restored. Reinitializing the tunneler/OIDC state by restarting the process was sufficient to recover.
Expected behavior
I would expect ziti-edge-tunnel to recover automatically when:
- Controller/network connectivity is lost.
- The API/OIDC session expires during the outage.
- Network connectivity is later restored.
If the previous API session can no longer be refreshed, I would expect the SDK to establish a new authenticated session rather than remain indefinitely in the unauthenticated state.
Possible reproduction
I believe this may be reproducible by:
- Start
ziti-edge-tunneland verify successful controller and edge-router connections. - Block network access to the controller and edge routers.
- Leave connectivity blocked long enough for the existing authentication/API session to expire.
- Restore network access.
- Observe whether the tunneler creates a new authenticated API session and reconnects to the edge routers.
In my test, the important transition occurred after roughly 27 minutes of outage, when the log changed from CONTROLLER_UNAVAILABLE to:
session token has expired
not authorized
no API session
Trying to re-auth
After that transition, the process never recovered on its own.