In the same environment, I attempted to use different agents, where both macOS and Windows could successfully expose services under the Tunnel network as gateways. However, when trying the Linux version of the Tunnel, I couldn't access it in any way. To address this, I tried switching between multiple cloud providers (such as AWS and Aliyun) for testing, but the issue persisted. How should I troubleshoot this? Because I haven't seen any useful error logs. Also, there shouldn't be any issues with the controller's configuration, as it works fine on Windows or macOS.
2 Likes
Hello and welcome to the OpenZiti discourse!
Could you please share the logs from ziti-edge-tunnel? The logs will be in different places depending on how you started it: if you start it from the command line then you see the logs on the terminal, and if you start it as a systemd service then you'll see the logs with journalctl -u ziti-edge-tunnel | cat.
Could you also show me the OpenZiti service(s) that you're trying to use with this ziti-edge-tunnel, as well as the related service policies?
My guess is that you need to give the ziti-edge-tunnel's identity bind permission for the service(s) that you're trying to use, but the logs and service info will help us narrow it down for certain.
Thanks!
2 Likes
Version information is as follows:
Kernel ๏ผ6.5.0-1014-aws #14~22.04.1-Ubuntu x86_64 GNU/Linux
Controller/Route: v0.32.2
ZAC: 2.9.4
ziti-edge-tunnel: v0.22.24
Mac ziti desktop edge: v0.22.24
ziti-edge-tunnel logs:
# journalctl -u ziti-edge-tunnel
Mar 05 03:42:35 ip-172-31-21-121 systemd[1]: Starting Ziti Edge Tunnel...
Mar 05 03:42:35 ip-172-31-21-121 ziti-edge-tunnel.sh[5416]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
Mar 05 03:42:35 ip-172-31-21-121 systemd[1]: Started Ziti Edge Tunnel.
Mar 05 03:42:35 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 0.000] ERROR ziti-edge-tunnel:instance-config.c:61 load_config_from_file() The config file No such file or directory cannot be opened due to /var/lib/ziti/config.json. This is normal if this is a new install or if the c>
Mar 05 03:42:35 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 0.000] ERROR ziti-edge-tunnel:instance-config.c:61 load_config_from_file() The config file No such file or directory cannot be opened due to /var/lib/ziti/config.json.backup. This is normal if this is a new install or i>
Mar 05 03:42:35 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 0.000] WARN ziti-edge-tunnel:instance-config.c:98 load_tunnel_status_from_file() Config files /var/lib/ziti/config.json and the backup file cannot be read or they do not exist, will create a new config file or the old >
Mar 05 03:42:35 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 0.064] ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
Mar 05 03:42:40 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 4.888] ERROR ziti-sdk:ziti_enroll.c:123 ziti_enroll() /github/workspace/build/_deps/ziti-sdk-c-src/library/ziti_enroll.c:108 - load_jwt_content(ecfg, &ecfg->zejh, &ecfg->zej) => -4 (JWT has invalid format)
Mar 05 03:42:40 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 4.888] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:319 tunnel_enroll_cb() enrollment failed: enroll failed(-4)
Mar 05 03:42:40 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 4.889] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:629 on_cmd() received from client - EOF. Closing connection.
Mar 05 03:42:40 ip-172-31-21-121 ziti-edge-tunnel[5417]: (5417)[ 4.889] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:641 on_cmd() IPC client connection closed, count: 0
Mar 05 03:42:52 ip-172-31-21-121 systemd[1]: ziti-edge-tunnel.service: Main process exited, code=dumped, status=11/SEGV
Mar 05 03:42:52 ip-172-31-21-121 systemd[1]: ziti-edge-tunnel.service: Failed with result 'core-dump'.
Mar 05 03:42:55 ip-172-31-21-121 systemd[1]: ziti-edge-tunnel.service: Scheduled restart job, restart counter is at 1.
Mar 05 03:42:55 ip-172-31-21-121 systemd[1]: Stopped Ziti Edge Tunnel.
Mar 05 03:42:55 ip-172-31-21-121 systemd[1]: Starting Ziti Edge Tunnel...
Mar 05 03:42:55 ip-172-31-21-121 systemd[1]: Started Ziti Edge Tunnel.
Mar 05 03:42:55 ip-172-31-21-121 ziti-edge-tunnel.sh[5456]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
Mar 05 03:42:55 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 0.000] WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/opt/openziti/etc/identities/HTTP Sever.json] is not loaded yet or already removed.
Mar 05 03:43:17 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 21.351] ERROR ziti-sdk:ziti_enroll.c:123 ziti_enroll() /github/workspace/build/_deps/ziti-sdk-c-src/library/ziti_enroll.c:108 - load_jwt_content(ecfg, &ecfg->zejh, &ecfg->zej) => -4 (JWT has invalid format)
Mar 05 03:43:17 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 21.351] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:319 tunnel_enroll_cb() enrollment failed: enroll failed(-4)
Mar 05 03:43:17 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 21.352] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:629 on_cmd() received from client - EOF. Closing connection.
Mar 05 03:43:17 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 21.352] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:641 on_cmd() IPC client connection closed, count: 0
Mar 05 03:44:34 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 98.638] ERROR ziti-sdk:ziti_enroll.c:123 ziti_enroll() /github/workspace/build/_deps/ziti-sdk-c-src/library/ziti_enroll.c:108 - load_jwt_content(ecfg, &ecfg->zejh, &ecfg->zej) => -4 (JWT has invalid format)
Mar 05 03:44:34 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 98.638] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:319 tunnel_enroll_cb() enrollment failed: enroll failed(-4)
Mar 05 03:44:34 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 98.639] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:629 on_cmd() received from client - EOF. Closing connection.
Mar 05 03:44:34 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 98.639] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:641 on_cmd() IPC client connection closed, count: 0
Mar 05 03:45:18 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 142.765] ERROR ziti-sdk:ziti_enroll.c:123 ziti_enroll() /github/workspace/build/_deps/ziti-sdk-c-src/library/ziti_enroll.c:108 - load_jwt_content(ecfg, &ecfg->zejh, &ecfg->zej) => -4 (JWT has invalid format)
Mar 05 03:45:18 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 142.765] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:319 tunnel_enroll_cb() enrollment failed: enroll failed(-4)
Mar 05 03:45:18 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 142.766] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:629 on_cmd() received from client - EOF. Closing connection.
Mar 05 03:45:18 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 142.766] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:641 on_cmd() IPC client connection closed, count: 0
Mar 05 03:45:39 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 164.068] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:629 on_cmd() received from client - EOF. Closing connection.
Mar 05 03:45:39 ip-172-31-21-121 ziti-edge-tunnel[5457]: (5457)[ 164.068] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:641 on_cmd() IPC client connection closed, count: 0
Mar 05 03:47:26 ip-172-31-21-121 systemd[1]: Stopping Ziti Edge Tunnel...
Mar 05 03:47:26 ip-172-31-21-121 systemd[1]: ziti-edge-tunnel.service: Deactivated successfully.
Mar 05 03:47:26 ip-172-31-21-121 systemd[1]: Stopped Ziti Edge Tunnel.
Mar 05 03:47:26 ip-172-31-21-121 systemd[1]: Starting Ziti Edge Tunnel...
Mar 05 03:47:26 ip-172-31-21-121 ziti-edge-tunnel.sh[5699]: (5699)[ 0.000] INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
Mar 05 03:47:26 ip-172-31-21-121 ziti-edge-tunnel.sh[5699]: (5699)[ 0.000] INFO ziti-sdk:utils.c:168 ziti_log_init() Ziti C SDK version 0.36.7 @11a8db3(HEAD) starting at (2024-03-05T03:47:26.105)
Mar 05 03:47:26 ip-172-31-21-121 ziti-edge-tunnel.sh[5699]: (5699)[ 0.000] INFO ziti-sdk:ziti_enroll.c:88 ziti_enroll() Ziti C SDK version 0.36.7 @11a8db3(HEAD) starting enrollment at (2024-03-05T03:47:26.105)
Mar 05 03:47:26 ip-172-31-21-121 ziti-edge-tunnel.sh[5695]: INFO: enrolled HTTP_Sever.jwt in /opt/openziti/etc/identities/HTTP_Sever.json
Mar 05 03:47:26 ip-172-31-21-121 systemd[1]: Started Ziti Edge Tunnel.
Mar 05 03:47:26 ip-172-31-21-121 ziti-edge-tunnel[5707]: (5707)[ 0.000] WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/opt/openziti/etc/identities/HTTP Server.json] is not loaded yet or already removed.
Mar 05 03:47:26 ip-172-31-21-121 ziti-edge-tunnel[5707]: (5707)[ 0.000] WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/opt/openziti/etc/identities/HTTP.json] is not loaded yet or already removed.
Mar 05 03:47:26 ip-172-31-21-121 ziti-edge-tunnel[5707]: (5707)[ 0.000] WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/opt/openziti/etc/identities/HTTP_Sever.json] is not loaded yet or already removed.
Mar 05 05:18:31 ip-172-31-21-121 ziti-edge-tunnel[5707]: (5707)[ 5465.527] WARN ziti-sdk:bind.c:192 session_cb() server[0.2] failed to get session for service[HTTP Service]: -29/NOT_FOUND
Mar 05 05:18:32 ip-172-31-21-121 ziti-edge-tunnel[5707]: (5707)[ 5466.332] WARN ziti-sdk:bind.c:192 session_cb() server[0.2] failed to get session for service[HTTP Service]: -29/NOT_FOUND
Mar 05 05:18:32 ip-172-31-21-121 ziti-edge-tunnel[5707]: (5707)[ 5466.332] ERROR tunnel-cbs:ziti_hosting.c:528 on_hosted_client_connect() hosted_service[HTTP Service] incoming connection failed: service not available
Mar 05 05:18:58 ip-172-31-21-121 ziti-edge-tunnel[5707]: (5707)[ 5492.256] WARN ziti-sdk:bind.c:192 session_cb() server[0.3] failed to get session for service[HTTP Service]: -29/NOT_FOUND
Mar 05 05:18:59 ip-172-31-21-121 systemd[1]: ziti-edge-tunnel.service: Main process exited, code=dumped, status=11/SEGV
Mar 05 05:18:59 ip-172-31-21-121 systemd[1]: ziti-edge-tunnel.service: Failed with result 'core-dump'.
Below is the log information from the controller:
Mar 7 04:19:26 ip-172-31-47-206 ziti[2195]: {"_context":"ch{RBepMJmcil}-\u003eu{classic}-\u003ei{Ra3W}","error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:74","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"RBepMJmcil","time":"2024-03-07T04:19:26.297Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273"}
Mar 7 04:19:26 ip-172-31-47-206 ziti[2231]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{nyAn}","chSeq":12,"connId":13,"edgeSeq":0,"error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:179","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-03-07T04:19:26.298Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273","type":"EdgeConnectType"}
Mar 7 04:19:26 ip-172-31-47-206 ziti[2195]: {"_context":"ch{RBepMJmcil}-\u003eu{classic}-\u003ei{Ra3W}","error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:74","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"RBepMJmcil","time":"2024-03-07T04:19:26.518Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273"}
Mar 7 04:19:26 ip-172-31-47-206 ziti[2231]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{nyAn}","chSeq":13,"connId":14,"edgeSeq":0,"error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:179","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-03-07T04:19:26.519Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273","type":"EdgeConnectType"}
Mar 7 04:19:26 ip-172-31-47-206 ziti[2195]: {"_context":"ch{RBepMJmcil}-\u003eu{classic}-\u003ei{Ra3W}","error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:74","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"RBepMJmcil","time":"2024-03-07T04:19:26.737Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273"}
Mar 7 04:19:26 ip-172-31-47-206 ziti[2231]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{nyAn}","chSeq":14,"connId":15,"edgeSeq":0,"error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:179","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-03-07T04:19:26.738Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273","type":"EdgeConnectType"}
Mar 7 04:19:27 ip-172-31-47-206 ziti[2195]: {"_context":"ch{RBepMJmcil}-\u003eu{classic}-\u003ei{Ra3W}","error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:74","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"RBepMJmcil","time":"2024-03-07T04:19:27.546Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273"}
Mar 7 04:19:27 ip-172-31-47-206 ziti[2231]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{nyAn}","chSeq":15,"connId":16,"edgeSeq":0,"error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:179","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-03-07T04:19:27.547Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273","type":"EdgeConnectType"}
Mar 7 04:19:27 ip-172-31-47-206 ziti[2195]: {"_context":"ch{RBepMJmcil}-\u003eu{classic}-\u003ei{Ra3W}","error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:74","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"RBepMJmcil","time":"2024-03-07T04:19:27.548Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273"}
Mar 7 04:19:27 ip-172-31-47-206 ziti[2231]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{nyAn}","chSeq":16,"connId":17,"edgeSeq":0,"error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:179","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-03-07T04:19:27.549Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273","type":"EdgeConnectType"}
Mar 7 04:19:32 ip-172-31-47-206 ziti[2195]: {"_context":"ch{RBepMJmcil}-\u003eu{classic}-\u003ei{Ra3W}","error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:74","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"RBepMJmcil","time":"2024-03-07T04:19:32.780Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273"}
Mar 7 04:19:32 ip-172-31-47-206 ziti[2231]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{nyAn}","chSeq":17,"connId":18,"edgeSeq":0,"error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:179","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-03-07T04:19:32.781Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273","type":"EdgeConnectType"}
Mar 7 04:19:32 ip-172-31-47-206 ziti[2195]: {"_context":"ch{RBepMJmcil}-\u003eu{classic}-\u003ei{Ra3W}","error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:74","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"RBepMJmcil","time":"2024-03-07T04:19:32.782Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273"}
Mar 7 04:19:32 ip-172-31-47-206 ziti[2231]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{nyAn}","chSeq":18,"connId":19,"edgeSeq":0,"error":"service 5NjcJOToxs5YFKGcceqYTX has no terminators","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:179","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-03-07T04:19:32.783Z","token":"8f28c678-fdbc-4923-ae1c-21a6a776b273","type":"EdgeConnectType"}
It's strange that with the old version installed last year (Controller/Route: v0.31.0), it can actually access normally.
I only switched the controller and the relevant router, but all versions of the agent remain the same. With the same configuration, everything works fine, so I feel the issue is not on the side of the Ziti edge tunnel.
1 Like
"no terminators" means your hosting identity was either not up, or not able to connect to a edge router.
can you list terminators from the controller. And also policy-advisor will help also.
1 Like
I feel there's no issue with the configuration because the same configuration works fine on older versions or on macOS and Windows.
root@ip-172-31-47-206:~# ziti edge list terminators # In the older versions, there is information here, but in the newly installed version, it's empty.
โญโโโโโฌโโโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโโฌโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฎ
โ ID โ SERVICE โ ROUTER โ BINDING โ ADDRESS โ IDENTITY โ COST โ PRECEDENCE โ DYNAMIC COST โ
โโโโโโผโโโโโโโโโโผโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโโโโโโโโค
โฐโโโโโดโโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโฏ
results: none
root@ip-172-31-47-206:~# ziti edge list identities
โญโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโโโโโโโโฎ
โ ID โ NAME โ TYPE โ ATTRIBUTES โ AUTH-POLICY โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโโโโโโโค
โ -DJWS-Yjqw โ zsf โ Default โ โ Default โ
โ 5RDYpVmPM โ Default Admin โ Default โ โ Default โ
โ RBepMJmcil โ ip-172-31-47-206-edge-router โ Router โ โ Default โ
โ Y4VQS-YKqw โ us โ Default โ โ Default โ
โ dNQgcQfjq โ hk โ Default โ โ Default โ
โฐโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโฏ
results: 1-5 of 5
root@ip-172-31-47-206:~# ziti edge list services
โญโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโฎ
โ ID โ NAME โ ENCRYPTION โ TERMINATOR STRATEGY โ ATTRIBUTES โ
โ โ โ REQUIRED โ โ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโค
โ 5NjcJOToxs5YFKGcceqYTX โ us-server โ true โ smartrouting โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโฏ
results: 1-1 of 1
root@ip-172-31-47-206:~# ziti edge list service-policies
โญโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโฎ
โ ID โ NAME โ SEMANTIC โ SERVICE ROLES โ IDENTITY ROLES โ POSTURE CHECK ROLES โ
โโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโค
โ 1htoqUiwcbiFqjYG7J6e41 โ us-server-dial โ AnyOf โ @us-server โ @zsf โ โ
โ 5RU5ee2i89zycHHfBZ6rpH โ us-server-bind โ AnyOf โ @us-server โ @us โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโฏ
results: 1-2 of 2
1 Like
Are we considering adding a variable for specifying the version in the expressInstall installation method?
1 Like
There is no information on the Terminators because terminators are not setup.
You can do:
ziti edge policy-advisor services
that should show you two lines for your service, one dial and one bind.
At beginning of that, it should say "OKAY", then the common routers should be at least 1.
you can also display the router status for us:
ziti fabric list routers
2 Likes
I initially thought it might be an issue with the AWS kernel being too high, so I redeployed an OpenZiti instance on Alibaba Cloud. Unfortunately, the result remains the same.
root@iZj6cfbiqa5u3hduq973olZ:~# ziti edge policy-advisor services -q
OKAY : hk (1) -> hk-server (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : zsf (1) -> hk-server (1) Common Routers: (1/1) Dial: Y Bind: N
root@iZj6cfbiqa5u3hduq973olZ:~# ziti fabric list routers
โญโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโฌโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ ID โ NAME โ ONLINE โ COST โ NO TRAVERSAL โ DISABLED โ VERSION โ LISTENERS โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ EYtg-9WhDH โ iZj6cfbiqa5u3hduq973olZ-edge-router โ true โ 0 โ false โ false โ v0.32.2 on linux/amd64 โ 1: tls:zt1.*****.xx:10080 โ
โฐโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
results: 1-1 of 1
root@iZj6cfbiqa5u3hduq973olZ:~#
1 Like
You have opened the port 10080 on the firewall ?
2 Likes
I did not turn on any firewall for testing, and the security group on the cloud also allowed all traffic. And I did a test verification of port 10080 from the outside, which was normal. In addition, I used mac as the tunnel gateway and I could see the normal "terminators". So I'm very curious why this is the case.
Here are some test information I did:
OpenZiti๏ผContoller/Route:๏ผ
root@iZj6cfbiqa5u3hduq973olZ:~# ziti edge policy-advisor services -q
OKAY : hk (1) -> hk-server (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : zsf (1) -> hk-server (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : hk (1) -> mac-server (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : zsf (1) -> mac-server (1) Common Routers: (1/1) Dial: Y Bind: Y
root@iZj6cfbiqa5u3hduq973olZ:~# ziti fabric list routers
โญโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโฌโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ ID โ NAME โ ONLINE โ COST โ NO TRAVERSAL โ DISABLED โ VERSION โ LISTENERS โ
โโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ EYtg-9WhDH โ iZj6cfbiqa5u3hduq973olZ-edge-router โ true โ 0 โ false โ false โ v0.32.2 on linux/amd64 โ 1: tls:zt1.***.com:10080 โ
โฐโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
results: 1-1 of 1
root@iZj6cfbiqa5u3hduq973olZ:~# ziti fabric list terminators
โญโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโฌโโโโโโโโโโโโฎ
โ ID โ SERVICE โ ROUTER โ BINDING โ ADDRESS โ INSTANCE โ COST โ PRECEDENCE โ DYNAMIC COST โ HOST ID โ
โโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโโโโโโโโผโโโโโโโโโโโโค
โ 4HfJthuD78DUFQpb9vaeWn โ mac-server โ iZj6cfbiqa5u3hduq973olZ-edge-router โ edge โ 4HfJthuD78DUFQpb9vaeWn โ โ 0 โ default โ 0 โ DV3Mj5mHl โ
โ gJxkPVNlsUdzDhgKs2Gzc โ mac-server โ iZj6cfbiqa5u3hduq973olZ-edge-router โ edge โ gJxkPVNlsUdzDhgKs2Gzc โ โ 0 โ default โ 0 โ DV3Mj5mHl โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโโโโโโโโดโโโโโโโโโโโโฏ
results: 1-2 of 2
Linux Client๏ผziti-edge-tunnel๏ผ
# telnet zt1.**.com 10080
Trying x.x.x.x...
Connected to zt1.xx.com.
Escape character is '^]'.
~# curl -i mac.xx.ziti
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.11.6
Date: Fri, 08 Mar 2024 04:41:30 GMT
Content-type: text/html; charset=utf-8
Content-Length: 838
mac server
MAC Client(Ziti Desktop Edge)
Mac# telnet zzt1.xx.com 10080
Trying 47.76.105.4...
Connected to zt1.xx.com.
Escape character is '^]'.
Mac# python3 -m http.server 8000
Serving HTTP on :: port 8000 (http://[::]:8000/) ...
The test access is the same, but mac----->hk is not accessible, but the other way around hk----->mac is accessible.
1 Like
The terminator gets created if the endpoint (hosting the service) can connect to a ER into the fabric. From your output, i assume you did an update on the service?
Since you verified the port 10080 is working, so that shouldn't be the issue.
So, the issue most likely is in the ziti-tunnel side. So i just looked carefully with the your ziti-edge-tunnel log, that is not running correctly.
(JWT has invalid format), did you create an Edge router or identity for this?
2 Likes
I suspect you may be running into a bug that was introduced with 0.32.2 and affects clients using the latest ziti-sdk-c (including ziti-edge-tunnel). Can you please check your edge router logs for messages like this?
INFO ziti/router/xgress_edge.(*edgeTerminator).close: {terminatorId=[1Bn5tvEtTos8WIki0s4k2Z] token=[388f6ead-1194-45cd-99b9-5ab13b67fd9b] reason=[terminator invalid]} removing terminator on controller
Thanks!
2 Likes
But when the latest version of Ziti Edge Tunnel is connected to the older version of the OpenZiti network, it works fine.
So, I feel like it's not an issue with Ziti Edge Tunnel. However, if it's related to the new version of the controller or router, my MAC or WIN client works fine. Anyway, when I did research testing last year with the old version, everything was normal. But this year, when I tried to deploy it in production, I encountered this issue. I'm not sure which part went wrong. At least I can guarantee that my configuration operations and some basic network settings should be fine.
OpenZiti๏ผContoller/Route๏ผ:
root@iZj6cfbiqa5u3hduq973olZ:~# cat /var/log/syslog*|grep -i 'xgress_edge'|grep -i removing|head
Mar 10 00:00:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"7FcPQlzw34IsP36Z7uLqiv","time":"2024-03-10T00:00:12.560Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:00:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"FVnAjjqLcvNOGUajH2bYf","time":"2024-03-10T00:00:12.561Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:05:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"3BLNJ4IRHEqoncjKfULnmV","time":"2024-03-10T00:05:12.589Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:05:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"7TocCdkEugXaA5QAPBCo36","time":"2024-03-10T00:05:12.590Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:10:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"56zT9enftAodqfpLeoTGnR","time":"2024-03-10T00:10:12.618Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:10:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"1fe8V9UsIuAeJ8Rxt5th4n","time":"2024-03-10T00:10:12.618Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:15:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"6bnhqe5kHeU27knIz46Izk","time":"2024-03-10T00:15:12.645Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:15:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"7dYfgBA2NLTk7nNNW6J4Dh","time":"2024-03-10T00:15:12.645Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:20:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"zKaI1f6Pl4w4uQcaVvWPx","time":"2024-03-10T00:20:12.673Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
Mar 10 00:20:12 iZj6cfbiqa5u3hduq973olZ ziti[25010]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:157","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).close","level":"info","msg":"removing terminator on controller","reason":"terminator invalid","terminatorId":"2TLKfIvvRdT9K6V7ohP5RO","time":"2024-03-10T00:20:12.674Z","token":"13cf36a7-9f4f-474a-89fd-e155105dd2f8"}
root@iZj6cfbiqa5u3hduq973olZ:~# cat /var/log/syslog*|grep -i 'xgress_edge'|grep -i removing|wc -l
2154
Linux Client๏ผziti-edge-tunnel๏ผ
root@iZj6ccikg76btt2lsaiw50Z:/var/log# cat /var/log/syslog*|grep -i terminator
Mar 6 11:35:40 iZj6ccikg76btt2lsaiw50Z ziti-edge-tunnel[8217]: (8217)[ 675.570] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.2/Gnu0ETfR/Connecting] failed to connect, reason=service Ril3mxCSfmkVWphkwC6ke has no terminators
Mar 6 11:36:48 iZj6ccikg76btt2lsaiw50Z ziti-edge-tunnel[12086]: (12086)[ 4.531] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.1/PCqoU3Ic/Connecting] failed to connect, reason=service Ril3mxCSfmkVWphkwC6ke has no terminators
Mar 6 11:56:24 iZj6ccikg76btt2lsaiw50Z ziti-edge-tunnel[19490]: (19490)[ 89.651] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.1/iqQDs2vp/Connecting] failed to connect, reason=service Ril3mxCSfmkVWphkwC6ke has no terminators
Mar 6 12:32:19 iZj6ccikg76btt2lsaiw50Z ziti-edge-tunnel[20803]: (20803)[ 116.807] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.1/XZM3iXcG/Connecting] failed to connect, reason=service Ril3mxCSfmkVWphkwC6ke has no terminators
Mar 6 12:32:50 iZj6ccikg76btt2lsaiw50Z ziti-edge-tunnel[20803]: (20803)[ 148.064] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.2/jR8hmLBk/Connecting] failed to connect, reason=service Ril3mxCSfmkVWphkwC6ke has no terminators
Mar 8 10:16:35 iZj6ccikg76btt2lsaiw50Z ziti-edge-tunnel[59265]: (59265)[ 64452.128] ERROR ziti-sdk:connect.c:963 connect_reply_cb() conn[0.2/U_Mk3WHL/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/2JeSDvv5x]: error creating route for [s/2JeSDvv5x] on [r/EYtg-9WhDH] (error creating route for [c/2JeSDvv5x]: failed to establish connection with terminator address 67szA3tS2TY1LlLBW2AByv. error: (rejected by application))
root@iZj6ccikg76btt2lsaiw50Z:/var/log# cat /var/log/syslog*|grep -i 'xgress_edge'
root@iZj6ccikg76btt2lsaiw50Z:/var/log#
2 Likes
Thanks for sharing the router logs. Those error messages make me more confident that your ziti-edge-tunnel is affected by the 0.32.2 changes. Your Mac and windows clients are using older versions of ziti-sdk-c, which are not affected by those changes. The fix for this issue should be released early this week. I'll post here when it's ready.
2 Likes
Thank you for your efforts, and may success follow you in the future.
1 Like
OpenZiti 0.33.0 includes the fix for the issue that prevents ziti-sdk-c clients from binding (hosting) services. The release is now available at Release v0.33.0 ยท openziti/ziti ยท GitHub
1 Like
Because I couldn't find a standard upgrade method, I had to rely on my understanding to perform the upgrade, which may not be conventional, but after testing, the previous issue has been resolved.
OpenZiti๏ผContoller/Route๏ผ:
# mkdir tmp
# wget -P tmp/ wget https://github.com/openziti/ziti/releases/download/v0.33.0/ziti-linux-amd64-0.33.0.tar.gz
# mkdir -pv ~/.ziti/quickstart/$(hostname -s)/ziti-bin/ziti-v0.33.0
# tar zxvf tmp/ziti-linux-amd64-0.33.0.tar.gz -C ~/.ziti/quickstart/$(hostname -s)/ziti-bin/ziti-v0.33.0
# sed -i 's/v0.32.2/v0.33.0/g' /etc/systemd/system/ziti-{controller,router}.service
# systemctl daemon-reload
# systemctl restart ziti-{controller,router}.service
# rm -rf tmp
Linux Client๏ผziti-edge-tunnel๏ผ
# apt update
# apt install ziti-edge-tunnel -y # update deb
# systemctl restart ziti-edge-tunnel.service
My MAC seems to be restarting frequently, and during the restart process, the icon is yellow. It's only accessible when the icon is green. Below are some related logs.
OpenZiti๏ผContoller/Route๏ผLogs:
Mar 14 11:17:14 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"_context":"tls:0.0.0.0:8441","error":"context deadline exceeded","file":"github.com/openziti/transport/v2@v2.0.122/tls/listener.go:228","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"58.33.29.26:50696","time":"2024-03-14T11:17:14.263Z"}
Mar 14 11:17:17 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:139","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).setState","level":"info","msg":"updated state","newState":2,"oldState":1,"reason":"replacing existing terminator","terminatorId":"2IPX94Kku8Frj1WKwe8mCy","time":"2024-03-14T11:17:17.726Z"}
Mar 14 11:17:17 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"file":"github.com/openziti/ziti/router/xgress_edge/fabric.go:139","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeTerminator).setState","level":"info","msg":"updated state","newState":3,"oldState":2,"reason":"newer terminator found for listener id","terminatorId":"2IPX94Kku8Frj1WKwe8mCy","time":"2024-03-14T11:17:17.726Z"}
Mar 14 11:17:17 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{dPwj}","connId":0,"existingTerminatorId":"2IPX94Kku8Frj1WKwe8mCy","file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:861","func":"github.com/openziti/ziti/router/xgress_edge.(*findMatchingEvent).handle","level":"info","listenerId":"\ufffd\ufffdc\ufffd\ufffdqp\ufffd7\ufffd\ufffd\ufffd7\ufffd[\ufffd\ufffdU\ufffd\u001a\ufffdE\ufffd\ufffdCฦฐr\ufffd\ufffd","msg":"taking over terminator from existing bind","routerId":"EYtg-9WhDH","terminatorId":"1BUqJKT5qk6QKufrz6cr07","time":"2024-03-14T11:17:17.726Z","token":"35d9bb39-b978-4754-95d0-99faa392fee9"}
Mar 14 11:17:17 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{dPwj}","bindConnId":0,"chSeq":12,"connId":0,"edgeSeq":0,"file":"github.com/openziti/ziti/router/xgress_edge/listener.go:486","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processBindV2","level":"info","listenerId":"\ufffd\ufffdc\ufffd\ufffdqp\ufffd7\ufffd\ufffd\ufffd7\ufffd[\ufffd\ufffdU\ufffd\u001a\ufffdE\ufffd\ufffdCฦฐr\ufffd\ufffd","msg":"sending replacement terminator success to sdk","routerId":"EYtg-9WhDH","terminatorId":"1BUqJKT5qk6QKufrz6cr07","time":"2024-03-14T11:17:17.726Z","token":"35d9bb39-b978-4754-95d0-99faa392fee9","type":"EdgeBindType"}
Mar 14 11:17:29 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"_context":"tls:0.0.0.0:8441","error":"context deadline exceeded","file":"github.com/openziti/transport/v2@v2.0.122/tls/listener.go:228","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"58.33.29.26:50751","time":"2024-03-14T11:17:29.340Z"}
Mar 14 11:17:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitId":"NBVb38J3N","ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":94858000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-03-14T11:17:46.574Z"}
Mar 14 11:17:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitCount":1,"ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:100","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","level":"warning","msg":"sent confirmation for circuits","time":"2024-03-14T11:17:46.574Z"}
Mar 14 11:17:46 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"circuitCount":1,"file":"github.com/openziti/ziti/controller/handler_ctrl/circuit_confirmation.go:46","func":"github.com/openziti/ziti/controller/handler_ctrl.(*circuitConfirmationHandler).HandleReceive","level":"info","msg":"received circuit confirmation request","routerId":"EYtg-9WhDH","time":"2024-03-14T11:17:46.574Z"}
Mar 14 11:17:54 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"_context":"tls:0.0.0.0:8441","error":"context deadline exceeded","file":"github.com/openziti/transport/v2@v2.0.122/tls/listener.go:228","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"58.33.29.26:50838","time":"2024-03-14T11:17:54.780Z"}
Mar 14 11:18:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitId":"NBVb38J3N","ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":154859000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-03-14T11:18:46.575Z"}
Mar 14 11:18:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitCount":1,"ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:100","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","level":"warning","msg":"sent confirmation for circuits","time":"2024-03-14T11:18:46.575Z"}
Mar 14 11:18:46 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"circuitCount":1,"file":"github.com/openziti/ziti/controller/handler_ctrl/circuit_confirmation.go:46","func":"github.com/openziti/ziti/controller/handler_ctrl.(*circuitConfirmationHandler).HandleReceive","level":"info","msg":"received circuit confirmation request","routerId":"EYtg-9WhDH","time":"2024-03-14T11:18:46.575Z"}
Mar 14 11:19:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitId":"NBVb38J3N","ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":214860000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-03-14T11:19:46.576Z"}
Mar 14 11:19:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitCount":1,"ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:100","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","level":"warning","msg":"sent confirmation for circuits","time":"2024-03-14T11:19:46.576Z"}
Mar 14 11:19:46 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"circuitCount":1,"file":"github.com/openziti/ziti/controller/handler_ctrl/circuit_confirmation.go:46","func":"github.com/openziti/ziti/controller/handler_ctrl.(*circuitConfirmationHandler).HandleReceive","level":"info","msg":"received circuit confirmation request","routerId":"EYtg-9WhDH","time":"2024-03-14T11:19:46.576Z"}
Mar 14 11:20:22 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"_context":"tls:0.0.0.0:8441","error":"context deadline exceeded","file":"github.com/openziti/transport/v2@v2.0.122/tls/listener.go:228","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"58.33.29.26:51309","time":"2024-03-14T11:20:22.689Z"}
Mar 14 11:20:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitId":"NBVb38J3N","ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":274860000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-03-14T11:20:46.576Z"}
Mar 14 11:20:46 iZj6cfbiqa5u3hduq973olZ ziti[181899]: {"circuitCount":1,"ctrlId":"iZj6cfbiqa5u3hduq973olZ","file":"github.com/openziti/ziti/router/forwarder/scanner.go:100","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","level":"warning","msg":"sent confirmation for circuits","time":"2024-03-14T11:20:46.576Z"}
Mar 14 11:20:46 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"circuitCount":1,"file":"github.com/openziti/ziti/controller/handler_ctrl/circuit_confirmation.go:46","func":"github.com/openziti/ziti/controller/handler_ctrl.(*circuitConfirmationHandler).HandleReceive","level":"info","msg":"received circuit confirmation request","routerId":"EYtg-9WhDH","time":"2024-03-14T11:20:46.577Z"}
Mar 14 11:20:48 iZj6cfbiqa5u3hduq973olZ ziti[181900]: {"_context":"tls:0.0.0.0:8441","error":"context deadline exceeded","file":"github.com/openziti/transport/v2@v2.0.122/tls/listener.go:228","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"58.33.29.26:51360","time":"2024-03-14T11:20:48.671Z"}
Linux Client๏ผziti-edge-tunnel๏ผLogs
root@iZj6ccikg76btt2lsaiw50Z:~# tail -f /var/log/syslog
Mar 14 11:25:40 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2216 of User root.
Mar 14 11:25:41 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2216.scope: Deactivated successfully.
Mar 14 11:25:41 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2217 of User root.
Mar 14 11:25:42 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2217.scope: Deactivated successfully.
Mar 14 11:25:44 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2218 of User root.
Mar 14 11:25:44 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2218.scope: Deactivated successfully.
Mar 14 11:25:45 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2219 of User root.
Mar 14 11:25:45 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2219.scope: Deactivated successfully.
Mar 14 11:25:46 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2220 of User root.
Mar 14 11:25:46 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2220.scope: Deactivated successfully.
Mar 14 11:25:50 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2221 of User root.
Mar 14 11:25:50 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2221.scope: Deactivated successfully.
Mar 14 11:25:51 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2222 of User root.
Mar 14 11:25:51 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2222.scope: Deactivated successfully.
Mar 14 11:25:52 iZj6ccikg76btt2lsaiw50Z systemd[1]: Started Session 2223 of User root.
Mar 14 11:25:52 iZj6ccikg76btt2lsaiw50Z systemd[1]: session-2223.scope: Deactivated successfully.
MAC Client Logs
excludeRoute() excludeRoute xx.xx.105.4 => xx.xx.105.4, 255.255.255.255
[2024-03-14T03:26:28:644Z] INFO PacketTunnelProvider:ZitiTunnelDelegate.swift:222 tunnelEventCallback() ZitiTunnelEvent: <CZiti.ZitiTunnelContextEvent: 0x7f96e845ae20>
identity: zsf:"DV3Mj5mHl"
status: OK
name: zsf
version: v0.33.0
controller: https://zt1.xxx:8441
code: 0
[2024-03-14T03:26:28:648Z] INFO PacketTunnelProvider:UserNotifications.swift:100 post() Attempting to post Info notification, subitile:Optional("Controller: Available"), body:Optional("zsf\nhttps://zt1.xxx:8441"), zid:Optional("zsf")
[2024-03-14T03:26:28:648Z] WARN PacketTunnelProvider:UserNotifications.swift:107 post() Not authorized to send notifications
(4552)[2024-03-14T03:38:51.635Z] INFO ziti-sdk:ziti.c:1561 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
I'd like to get a better look at what's going on with your Mac client. Can you please share the full appex.log? You can email it to help@openziti.org or DM a file sharing link to me here.
Hello, I have sent the appex.log file through my GMAIL. Please check. Thank you!
Thanks! You can see the connections to controller are flapping. One attempt fails:
2024-03-15T01:28:45.543Z] ERROR ziti-sdk:ziti.c:1100 update_services() ztx[0] failed to get service updates err[CONTROLLER_UNAVAILABLE/software caused connection abort] from ctrl[https://zt1.xxx:8441]
[2024-03-15T01:28:45.543Z] WARN tunnel-cbs:ziti_tunnel_ctrl.c:781 on_ziti_event() ziti_ctx controller connections failed: ziti controller is not available
And the very next attempt just 10 seconds later succeeds:
[2024-03-15T01:28:56.552Z] INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[zsf] connected to controller
And the next request fails as above, and so on... until the end of the log, where it looks like you changed the log level to debug. From that point on we see several successful connections with the controller, and no more errors - although it is only the last few minutes of the log so maybe the connections started failing again?
I'm not sure if those connection failures relate to the ssl handshake failures that you see in the controller logs. Did this start happening when you updated the controller to 0.33.0?
I noticed your Mac client is at version 2.37, a couple of revs behind. Do you still see these problems after updating to 2.39?