Ziti tunnel stopped talking to controller

Tunneler Apps Ziti Tunnel
1

I’ve had an interesting fault occur, we’ve seen issues where routers crash or become unavailable in the past and restarts have fixed things however we’ve run into an issue now with the 1.9.16 tunnelers

We’ve been running 1.6.12 on the controller and in the past batch of rpi’s that got sent out we have 1.8.3 on the tunneler which have been running fine, the next batch went out with updates and are running 1.9.16 and after a router crash most of them didn’t come back. Most of these are in the wild so we couldn’t get to them but 1 was and we’ve seen this in the log:

Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ZitiControllerURL:1280] request[/current-identity] failed: -3008(unknown node or service)
Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] ERROR ziti-sdk:ziti.c:1568 update_identity_data() ztx[1] failed to get identity_data: unknown node or service[CONTROLLER_UNAVAILABLE]
Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ZitiControllerURL:1280] request[/current-identity/edge-routers?limit=25&offset=0] failed: -3008(unknown node o)
Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] ERROR ziti-sdk:ziti.c:1491 edge_routers_cb() ztx[1] failed to get current edge routers: code[0] CONTROLLER_UNAVAILABLE/unknown node or service
Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] WARN ziti-sdk:ziti_ctrl.c:177 ctrl_resp_cb() ctrl[https://ZitiControllerURL:1280] request[/current-api-session/service-updates] failed: -3008(unknown node or service)
Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] INFO ziti-sdk:ziti_ctrl.c:180 ctrl_resp_cb() ctrl[https://ZitiControllerURL:1280] attempting to switch endpoint
Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] WARN ziti-sdk:ziti_ctrl.c:604 ctrl_next_ep() ctrl[https://ZitiControllerURL:1280] no controllers are online
Feb 05 15:17:19 rpi ziti-edge-tunnel[743]: (743)[ 187316.000] WARN ziti-sdk:ziti.c:1439 check_service_update() ztx[1] failed to poll service updates: code[0] err[-16/unknown node or service]
Feb 05 15:17:22 rpi ziti-edge-tunnel[743]: (743)[ 187318.698] ERROR tlsuv:http.c:354 http[ZitiControllerURL:1280/oidc/oauth/token]: connection failed: unknown node or service
Feb 05 15:17:22 rpi ziti-edge-tunnel[743]: (743)[ 187318.698] WARN ziti-sdk:oidc.c:622 refresh_cb() oidc[internal] OIDC token refresh failed (trying again): -3008/unknown node or service

A simple restart via “systemctrl restart ziti-edge-tunnel” solved it but we’ve got a bunch of devices we’re going to have to get rebooted. Is there anything known about this sort of error? i’ve checked the device and there aren’t any more updates to the edge tunnel

2

Hi @BeccaraNZ , I can't help with the tunneler side, but if you have more information on why the routers are crashing, I'd be very interested. Is it panics or out-of-memory or something else? Let me know if you've got anything you can share.

Cheers,
Paul

3

I too would love to know, traffic just stops routing and tbh trying to work thru logging in the middle of the event isn’t flash with the way the routers log into journal. I’ll work on capturing a slice to post up here next time it happens

1 Like
4

If it's no longer responding but still running then stack dumps are often the most revealing. If it's feasible, see How To Gather OpenZiti Diagnostics · openziti/ziti Wiki · GitHub for some info on how to gather them.

Thank you!
Paul