Terminators and service issue

1

Hi,
I hope you guys are well.

Suddenly client (windows) can't access a service. If I recall, terminator was not created so last time I just restarted ziti-router service.

So now I restarted services (ziti-controller and ziti-router in openziti server and even whole system) as this issue just came today. Here the logs (OpenZiti and Client).

Web server (Ubuntu 18.04.4 LTS)

systemctl status ziti-edge-tunnel
● ziti-edge-tunnel.service - Ziti Edge Tunnel
   Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ziti-edge-tunnel.service.d
           └─10-run-as-root.conf
   Active: active (running) since Mon 2023-10-16 15:06:54 +03; 46s ago
  Process: 21582 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
 Main PID: 21584 (ziti-edge-tunne)
    Tasks: 5 (limit: 4657)
   CGroup: /system.slice/ziti-edge-tunnel.service
           └─21584 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

16 15:06:54 odoo-prod systemd[1]: Started Ziti Edge Tunnel.
16 15:06:54 odoo-prod ziti-edge-tunnel[21584]: (21584)[        0.085]    WARN ziti-edge-tunnel:resolvers.c:317 set_systemd_resolved_link_setting() Attempted to call unknown method: SetLinkDNSOverTLS for link: (ziti0)
16 15:07:09 odoo-prod ziti-edge-tunnel[21584]: (21584)[       15.097]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[ztn.s-apps.io] request failed: -110(connection timed out)
16 15:07:09 odoo-prod ziti-edge-tunnel[21584]: (21584)[       15.097]   ERROR ziti-sdk:ziti.c:1515 version_cb() ztx[0] failed to get controller version from https://ztn.s-apps.io:8441 CONTROLLER_UNAVAILABLE(connection timed out)
16 15:07:09 odoo-prod ziti-edge-tunnel[21584]: (21584)[       15.097]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[ztn.s-apps.io] request failed: -110(connection timed out)
16 15:07:09 odoo-prod ziti-edge-tunnel[21584]: (21584)[       15.097]    WARN ziti-sdk:ziti.c:1444 api_session_cb() ztx[0] failed to get api session from ctrl[https://ztn.s-apps.io:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] connection timed out
16 15:07:09 odoo-prod ziti-edge-tunnel[21584]: (21584)[       15.097]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:744 on_ziti_event() ziti_ctx controller connections failed: ziti controller is not available
16 15:07:09 odoo-prod ziti-edge-tunnel[21584]: (21584)[       15.097]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1199 on_event() ztx[/opt/openziti/etc/identities/odoo-prod-pc.json] failed to connect to controller due to ziti controller is not available
16 15:07:29 odoo-prod ziti-edge-tunnel[21584]: (21584)[       35.098]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[ztn.s-apps.io] request failed: -110(connection timed out)
16 15:07:29 odoo-prod ziti-edge-tunnel[21584]: (21584)[       35.098]    WARN ziti-sdk:ziti.c:1444 api_session_cb() ztx[0] failed to get api session from ctrl[https://ztn.s-apps.io:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] connection timed out

In OpenZiti (router and controller. Ubuntu 22.04.3 LTS)

systemctl status ziti-router.service
● ziti-router.service - Ziti-Router for ztn-edge-router
     Loaded: loaded (/etc/systemd/system/ziti-router.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-10-16 11:49:52 UTC; 20min ago
   Main PID: 846 (ziti)
      Tasks: 8 (limit: 6970)
     Memory: 52.9M
        CPU: 4.075s
     CGroup: /system.slice/ziti-router.service
             └─846 /root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0/ziti router run /root/.ziti/quickstart/ztn/ztn-edge-router.yaml

Oct 16 12:10:20 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":171,"connId":340,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:20.090Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:20 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":172,"connId":341,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:20.354Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:20 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":173,"connId":342,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:20.606Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:20 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":174,"connId":343,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:20.868Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:21 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":175,"connId":344,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:21.123Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:21 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":176,"connId":345,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:21.392Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:21 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":177,"connId":346,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:21.639Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:21 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":178,"connId":347,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:21.909Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:22 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":179,"connId":348,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:22.158Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}
Oct 16 12:10:22 ztn ziti[846]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{Go3G}","chSeq":180,"connId":349,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-10-16T12:10:22.425Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8","type":"EdgeConnectType"}

Controller

systemctl status ziti-controller.service
● ziti-controller.service - Ziti-Controller
     Loaded: loaded (/etc/systemd/system/ziti-controller.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-10-16 11:49:52 UTC; 22min ago
   Main PID: 844 (ziti)
      Tasks: 9 (limit: 6970)
     Memory: 111.1M
        CPU: 17.862s
     CGroup: /system.slice/ziti-controller.service
             └─844 /root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0/ziti controller run /root/.ziti/quickstart/ztn/ztn.yaml

Oct 16 12:10:20 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:20.090Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:20 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:20.353Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:20 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:20.606Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:20 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:20.868Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:21 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:21.122Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:21 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:21.391Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:21 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:21.639Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:21 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:21.909Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:22 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:22.157Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}
Oct 16 12:10:22 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{7vwb}","error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-10-16T12:10:22.424Z","token":"4297dbf8-2231-41c3-aabf-3b1d9c43a5b8"}

Terminators (2 are missing. one is the above and another is to forward traffic)

ziti edge list terminators
╭────────────────────────┬─────────────┬─────────────────┬─────────┬─────────────────────────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID                     │ SERVICE     │ ROUTER          │ BINDING │ ADDRESS                                     │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├────────────────────────┼─────────────┼─────────────────┼─────────┼─────────────────────────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ 4ABscVZDTKfiH1YgtORTva │ journal2    │ ztn-edge-router │ edge    │ hosted:5354ecc6-134e-4bf5-99be-7a6269e64d20 │          │    0 │ default    │            0 │
│ 5CchqEkHfXvjmE639eOK3h │ odoo-web    │ ztn-edge-router │ edge    │ hosted:f096042c-9a99-49ce-ad34-baae1a3a95f0 │          │    0 │ default    │            0 │
│ 5baNc1NnkiAqUKTOrxXc2I │ journal     │ ztn-edge-router │ edge    │ hosted:683516ee-9778-4fd1-85ea-bcbc1a21aa13 │          │    0 │ default    │            0 │
│ 5e3OaVv5MuEMIfA40fAIIv │ connectvas  │ ztn-edge-router │ edge    │ hosted:78a69160-2a7c-4b87-946d-e5a31664138b │          │    0 │ default    │            0 │
│ 6KfZ9fC1joHKtSBIdY9boJ │ connectvas2 │ ztn-edge-router │ edge    │ hosted:ae0777a0-3e4f-48b4-8eaa-1bed71029dc4 │          │    0 │ default    │            0 │
╰────────────────────────┴─────────────┴─────────────────┴─────────┴─────────────────────────────────────────────┴──────────┴──────┴────────────┴──────────────╯

Thank you

2

UPDATE: After restarting 2nd time. The service is accessible and terminators are created that were missing.

ziti edge list terminators
╭──────────────────────────────────────┬────────────────┬─────────────────┬─────────┬─────────────────────────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ 5r7RBUlKD89BvaAxvr4l3L               │ odooprod       │ ztn-edge-router │ edge    │ hosted:373ed5c4-68f2-436a-bd14-8887df02abe4 │          │    0 │ default    │            0 │
│ 93b1c6aa-cfe0-4667-85e0-fb30a41249e9 │ myweb          │ ztn-edge-router │ tunnel  │ 93b1c6aa-cfe0-4667-85e0-fb30a41249e9        │          │    0 │ default    │            0 │
╰──────────────────────────────────────┴────────────────┴─────────────────┴─────────┴─────────────────────────────────────────────┴──────────┴──────┴────────────┴──────────────╯
results: 1-8 of 8
3

The web server with the Tunneler is saying it can't reach the controller, so of course it can't build a terminator in the router. When you see the controller not available, you should look to see why that is. Normally, restarting the controller shouldn't change that, unless it is just a timing issue. You can check the path, by using curl to see if there is a clear path.

curl -k https://<SERVER NAME OR IP>:PORT/version

You can obviously make sure the controller process is running and bound to the correct port, etc. if that fails. If the process appears to be running, you can curl locally on the controller to see if the port is responding properly, then you know the problem is between the tunneler and the controller.

1 Like
4

Hi Mike Gorman,
Thank you very much for your response.
This issue arrived before (Send traffic to (without edge client in server) - #7 by kashif) and , so i restarted the "ziti-router.service" service and then it worked fine.
Next time I will follow the check you have told.
Thanks again.