Hi Again,
I've been building my backup/restore process and i've come across an issue in Ziti Edge Tunnel which i'm only able to recover from by restarting the ZET service.
Firstly i establish a Ziti network with 3 HA Controllers and 2 public Edge Routers using v1.5.4. I create a number of test identities, enrol them using ZET v1.5.10 and leave them running.
To back up my system i take a copy of the PKI root CA dir at the time it's first created. At the time of backup creation i take copies of config.yml and trigger a DB snapshot.
Now i destroy the VM's my infrastructure has been running on.
Next i create a new Debian VM to act as a bootstrap Controller. I restore the backed up Configuration with the following basic steps.
1). Using the backed up root CA, create new PKI for the other HA Controllers.
2). Restore the backed up config.yml to /var/lib/private/ziti-controller/
3). Restore the DB ziti agent cluster restore-from-db /path/to/backup.db
At this point, i've got a single HA Controller with my previously backed up configuration. I simply add other Controllers and Edge Routers as if they were new components.
While i was backing up, destroying and restoring my Ziti infrastructure my ZET clients were busy attempting to re-connect to the network.
However once normal infrastructure service is resumed, the ZET client doesn't seem to connect and i see the following repeating in the logs. (verbose level 3)
(64)[2025-04-17T13:49:56.554Z] ERROR ziti-sdk:ziti.c:1480 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] UNAUTHORIZED/no api session token set for ziti_controller
(64)[2025-04-17T13:49:56.554Z] WARN ziti-sdk:ziti_ctrl.c:804 verify_api_session() ctrl[ziti-controller-1.az.lifeboat.ziti:443] no API session
(64)[2025-04-17T13:49:56.554Z] WARN ziti-sdk:ziti.c:1428 check_service_update() ztx[0] failed to poll service updates: code[0] err[-14/no api session token set for ziti_controller]
(64)[2025-04-17T13:50:06.507Z] INFO ziti-sdk:channel.c:772 reconnect_cb() ch[0] ziti context is not fully authenticated (auth_state[0]), delaying re-connect
(64)[2025-04-17T13:50:06.507Z] INFO ziti-sdk:channel.c:819 reconnect_channel() ch[0] reconnecting in 85075ms (attempt = 35)
(64)[2025-04-17T13:50:29.237Z] INFO ziti-sdk:channel.c:772 reconnect_cb() ch[1] ziti context is not fully authenticated (auth_state[0]), delaying re-connect
(64)[2025-04-17T13:50:29.237Z] INFO ziti-sdk:channel.c:819 reconnect_channel() ch[1] reconnecting in 9760ms (attempt = 36)
(64)[2025-04-17T13:50:38.997Z] INFO ziti-sdk:channel.c:772 reconnect_cb() ch[1] ziti context is not fully authenticated (auth_state[0]), delaying re-connect
(64)[2025-04-17T13:50:38.997Z] INFO ziti-sdk:channel.c:819 reconnect_channel() ch[1] reconnecting in 2517ms (attempt = 37)
(64)[2025-04-17T13:50:41.515Z] INFO ziti-sdk:channel.c:772 reconnect_cb() ch[1] ziti context is not fully authenticated (auth_state[0]), delaying re-connect
(64)[2025-04-17T13:50:41.515Z] INFO ziti-sdk:channel.c:819 reconnect_channel() ch[1] reconnecting in 41686ms (attempt = 38)
(64)[2025-04-17T13:50:56.554Z] WARN ziti-sdk:ziti_ctrl.c:804 verify_api_session() ctrl[ziti-controller-1.az.lifeboat.ziti:443] no API session
(64)[2025-04-17T13:50:56.554Z] ERROR ziti-sdk:ziti.c:1555 update_identity_data() ztx[0] failed to get identity_data: no api session token set for ziti_controller[UNAUTHORIZED]
(64)[2025-04-17T13:50:56.554Z] WARN ziti-sdk:ziti_ctrl.c:804 verify_api_session() ctrl[ziti-controller-1.az.lifeboat.ziti:443] no API session
(64)[2025-04-17T13:50:56.554Z] ERROR ziti-sdk:ziti.c:1480 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] UNAUTHORIZED/no api session token set for ziti_controller
(64)[2025-04-17T13:50:56.554Z] WARN ziti-sdk:ziti_ctrl.c:804 verify_api_session() ctrl[ziti-controller-1.az.lifeboat.ziti:443] no API session
(64)[2025-04-17T13:50:56.554Z] WARN ziti-sdk:ziti.c:1428 check_service_update() ztx[0] failed to poll service updates: code[0] err[-14/no api session token set for ziti_controller]
(64)[2025-04-17T13:51:23.203Z] INFO ziti-sdk:channel.c:772 reconnect_cb() ch[1] ziti context is not fully authenticated (auth_state[0]), delaying re-connect
(64)[2025-04-17T13:51:23.203Z] INFO ziti-sdk:channel.c:819 reconnect_channel() ch[1] reconnecting in 44702ms (attempt = 39)
(64)[2025-04-17T13:51:31.582Z] INFO ziti-sdk:channel.c:772 reconnect_cb() ch[0] ziti context is not fully authenticated (auth_state[0]), delaying re-connect
(64)[2025-04-17T13:51:31.582Z] INFO ziti-sdk:channel.c:819 reconnect_channel() ch[0] reconnecting in 147321ms (attempt = 36)
(64)[2025-04-17T13:51:56.554Z] WARN ziti-sdk:ziti_ctrl.c:804 verify_api_session() ctrl[ziti-controller-1.az.lifeboat.ziti:443] no API session
(64)[2025-04-17T13:51:56.554Z] ERROR ziti-sdk:ziti.c:1555 update_identity_data() ztx[0] failed to get identity_data: no api session token set for ziti_controller[UNAUTHORIZED]
(64)[2025-04-17T13:51:56.554Z] WARN ziti-sdk:ziti_ctrl.c:804 verify_api_session() ctrl[ziti-controller-1.az.lifeboat.ziti:443] no API session
(64)[2025-04-17T13:51:56.554Z] ERROR ziti-sdk:ziti.c:1480 edge_routers_cb() ztx[0] failed to get current edge routers: code[0] UNAUTHORIZED/no api session token set for ziti_controller
(64)[2025-04-17T13:51:56.554Z] WARN ziti-sdk:ziti_ctrl.c:804 verify_api_session() ctrl[ziti-controller-1.az.lifeboat.ziti:443] no API session
(64)[2025-04-17T13:51:56.554Z] WARN ziti-sdk:ziti.c:1428 check_service_update() ztx[0] failed to poll service updates: code[0] err[-14/no api session token set for ziti_controller]
If i manually restart the ZET service systemctl restart ziti-edge-tunnel.service It connects back to the infrastructure straight away.