Slow response and sometimes website inaccessible

kashif · December 5, 2023, 11:21am

Hi,
I hope you all are doing well.

I installed an edge client on another web server, details are below

 IP: 192.168.20.x 
 OS: Ubuntu 18LTS
 URL: erpaman.ziti:8069

The response of the web "erpaman.ziti" is slow whereas if accessing "erp.aman.lc" without ziti tunnel it's normal. The difference is around 10 seconds but sometimes it takes more time to load.

Note: OpenZiti Server IP 192.168.17.x OS: Ubuntu 22LTS. Controller: v0.30.0 ZAC: 2.8.7

Here's the output of the edge client on web server

Before restart service. (There were more than1 terminators for edge client.)

● ziti-edge-tunnel.service - Ziti Edge Tunnel
   Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ziti-edge-tunnel.service.d
           └─10-run-as-root.conf
   Active: active (running) since Tue 2023-12-05 12:59:10 +03; 3min 18s ago
  Process: 2062 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
 Main PID: 2063 (ziti-edge-tunne)
    Tasks: 9 (limit: 4915)
   CGroup: /system.slice/ziti-edge-tunnel.service
           └─2063 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

 05 12:59:10 odoo-prod systemd[1]: Starting Ziti Edge Tunnel...
 05 12:59:10 odoo-prod ziti-edge-tunnel.sh[2062]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
 05 12:59:10 odoo-prod systemd[1]: Started Ziti Edge Tunnel.
 05 12:59:10 odoo-prod ziti-edge-tunnel[2063]: (2063)[        0.042]    WARN ziti-edge-tunnel:resolvers.c:317 set_systemd_resolved_link_setting() Attempted to call unknown method: SetLinkDNSOverTLS for link: (ziti0)
 05 13:00:29 odoo-prod ziti-edge-tunnel[2063]: (2063)[       78.472]    WARN ziti-sdk:channel.c:337 on_channel_send() ch[0] write delay = 5.231 q=2 qs=19443
 05 13:00:35 odoo-prod ziti-edge-tunnel[2063]: (2063)[       84.804]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.5] closing bridge due to error: -21(connection to edge router terminated)
 05 13:00:35 odoo-prod ziti-edge-tunnel[2063]: (2063)[       84.804]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.3] closing bridge due to error: -21(connection to edge router terminated)
 05 13:00:35 odoo-prod ziti-edge-tunnel[2063]: (2063)[       84.804]    WARN ziti-sdk:bind.c:346 on_message() binding failed: -21/connection to edge router terminated
 05 13:00:44 odoo-prod ziti-edge-tunnel[2063]: (2063)[       93.659]   ERROR ziti-sdk:channel.c:637 hello_reply_cb() ch[0] failed to receive Hello response due to -21(connection to edge router terminated)

After restart service

● ziti-edge-tunnel.service - Ziti Edge Tunnel
   Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ziti-edge-tunnel.service.d
           └─10-run-as-root.conf
   Active: active (running) since Tue 2023-12-05 13:02:58 +03; 21min ago
  Process: 2612 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
 Main PID: 2613 (ziti-edge-tunne)
    Tasks: 9 (limit: 4915)
   CGroup: /system.slice/ziti-edge-tunnel.service
           └─2613 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

 05 13:05:54 odoo-prod ziti-edge-tunnel[2613]: (2613)[      175.382]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.14] closing bridge due to error: -21(connection to edge router terminated)
 05 13:05:54 odoo-prod ziti-edge-tunnel[2613]: (2613)[      175.382]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.13] closing bridge due to error: -21(connection to edge router terminated)
 05 13:05:54 odoo-prod ziti-edge-tunnel[2613]: (2613)[      175.382]    WARN ziti-sdk:bind.c:346 on_message() binding failed: -21/connection to edge router terminated
 05 13:05:57 odoo-prod ziti-edge-tunnel[2613]: (2613)[      178.800]   ERROR ziti-sdk:channel.c:637 hello_reply_cb() ch[0] failed to receive Hello response due to -21(connection to edge router terminated)
 05 13:07:56 odoo-prod ziti-edge-tunnel[2613]: (2613)[      297.181]    WARN ziti-sdk:channel.c:337 on_channel_send() ch[0] write delay = 5.765 q=2 qs=19426
 05 13:08:19 odoo-prod ziti-edge-tunnel[2613]: (2613)[      320.089]   ERROR ziti-sdk:channel.c:595 latency_timeout() ch[0] no read/write traffic on channel since before latency probe was sent, closing channel
 05 13:08:19 odoo-prod ziti-edge-tunnel[2613]: (2613)[      320.089]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.24] closing bridge due to error: -20(operation did not complete in time)
 05 13:08:19 odoo-prod ziti-edge-tunnel[2613]: (2613)[      320.089]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.22] closing bridge due to error: -20(operation did not complete in time)
 05 13:08:19 odoo-prod ziti-edge-tunnel[2613]: (2613)[      320.089]    WARN ziti-sdk:bind.c:346 on_message() binding failed: -20/operation did not complete in time
 05 13:12:00 odoo-prod ziti-edge-tunnel[2613]: (2613)[      541.321]    WARN ziti-sdk:channel.c:337 on_channel_send() ch[0] write delay = 5.787 q=2 qs=19147

Thank you for your support

kashif · December 6, 2023, 7:59am

UPDATE:
After some time, I saw 2 terminators of this service (and page in not available/accessible) and sometimes its showing more. The output of the service is below

root@odoo-prod:/home/odoouser# systemctl status ziti-edge-tunnel.service
● ziti-edge-tunnel.service - Ziti Edge Tunnel
   Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ziti-edge-tunnel.service.d
           └─10-run-as-root.conf
   Active: active (running) since Tue 2023-12-05 15:04:20 +03; 3min 10s ago
  Process: 19576 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
 Main PID: 19577 (ziti-edge-tunne)
    Tasks: 9 (limit: 4915)
   CGroup: /system.slice/ziti-edge-tunnel.service
           └─19577 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

 05 15:04:20 odoo-prod systemd[1]: Starting Ziti Edge Tunnel...
 05 15:04:20 odoo-prod ziti-edge-tunnel.sh[19576]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
 05 15:04:20 odoo-prod systemd[1]: Started Ziti Edge Tunnel.
 05 15:04:20 odoo-prod ziti-edge-tunnel[19577]: (19577)[        0.039]    WARN ziti-edge-tunnel:resolvers.c:317 set_systemd_resolved_link_setting() Attempted to call unknown method: SetLinkDNSOverTLS for link: (ziti0)
 05 15:05:30 odoo-prod ziti-edge-tunnel[19577]: (19577)[       70.488]   ERROR ziti-sdk:channel.c:595 latency_timeout() ch[0] no read/write traffic on channel since before latency probe was sent, closing channel
 05 15:05:30 odoo-prod ziti-edge-tunnel[19577]: (19577)[       70.488]    WARN ziti-sdk:bind.c:346 on_message() binding failed: -20/operation did not complete in time
 05 15:05:30 odoo-prod ziti-edge-tunnel[19577]: (19577)[       70.488]    WARN ziti-sdk:channel.c:337 on_channel_send() ch[0] write delay = 61.110 q=3 qs=19603
 05 15:05:30 odoo-prod ziti-edge-tunnel[19577]: (19577)[       70.488]   ERROR ziti-sdk:channel.c:357 on_channel_send() ch[0] write failed [-125/operation canceled]

scareything · December 6, 2023, 1:08pm

Hello,

Which version of ziti-edge-tunnel is this? There are some issues with 0.22.15 that we are working on which could explain what you are seeing. Can you get the logs from the edge router?

Thanks,
-Shawn

kashif · December 6, 2023, 1:29pm

Thank you for your reply.

Okay I get it now. Anyway, here's the information you asked for.

On client (webserver)

ziti-edge-tunnel version
v0.22.15-local

On Openziti

journalctl -u ziti-controller --since "10min ago"

Dec 06 13:14:44 ztn ziti[844]: {"file":"github.com/openziti/fabric@v0.24.2/controller/network/fault.go:31","func":"github.com/openziti/fabric/controller/network.(*Network).fault","level":"info","msg":"network fault processing for [1] circuits","time":"2023-12-06T13:14:44.253Z"}
Dec 06 13:14:44 ztn ziti[844]: {"circuitId":"aLpn3r0TB","file":"github.com/openziti/fabric@v0.24.2/controller/network/fault.go:48","func":"github.com/openziti/fabric/controller/network.(*Network).fault","level":"info","msg":"sent unroute for circuit to router in response to forwarding fault","routerId":"vQutsJRrHa","time":"2023-12-06T13:14:44.253Z"}
Dec 06 13:16:29 ztn ziti[844]: {"file":"github.com/openziti/fabric@v0.24.2/controller/network/fault.go:31","func":"github.com/openziti/fabric/controller/network.(*Network).fault","level":"info","msg":"network fault processing for [1] circuits","time":"2023-12-06T13:16:29.257Z"}
Dec 06 13:16:29 ztn ziti[844]: {"circuitId":"Wlbw3rVTB","file":"github.com/openziti/fabric@v0.24.2/controller/network/fault.go:48","func":"github.com/openziti/fabric/controller/network.(*Network).fault","level":"info","msg":"sent unroute for circuit to router in response to forwarding fault","routerId":"vQutsJRrHa","time":"2023-12-06T13:16:29.257Z"}
Dec 06 13:17:48 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:48.234Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:17:48 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:48.390Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:17:48 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:48.740Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:17:48 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:48.909Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:17:49 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:49.254Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:17:49 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:49.425Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:17:49 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:49.533Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:17:49 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:17:49.536Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
.....
Dec 06 13:23:44 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:23:44.149Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:23:44 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:23:44.396Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:23:44 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:23:44.665Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}
Dec 06 13:23:44 ztn ziti[844]: {"_context":"ch{vQutsJRrHa}-\u003eu{classic}-\u003ei{y4k9}","error":"service 4cYpWQrSzYi0kpeFNUHjC6 has no terminators","file":"github.com/openziti/edge@v0.24.381/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/edge/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"vQutsJRrHa","time":"2023-12-06T13:23:44.916Z","token":"19dfb86b-d7ef-4c85-94d5-a390cf6b388c"}

Thank you very much

scareything · December 6, 2023, 1:52pm

Ok! That sort of sounds like your service is working well now, but let me know if you're still having issues.

The errors in your router log make me suspect that the hosting tunneler (or edge router) for your service either isn't (or wasn't) started or is/was having trouble. If you are still having issues, what is running the other side of this service? Another ziti-edge-tunnel perhaps? Can you share the logs from the other end of the ziti connection?

Thanks,
-Shawn

kashif · December 10, 2023, 10:31am

Still having issues. After restart it works and after some time no response, there's no exact timeline for stoppage but I'll keeping monitoring the web (erpaman.ziti:8069), please note that it's slow to respond too.

In openziti server (note, I have restarted ziti-controller.service and ziti-router.service services)

● ziti-router.service - Ziti-Router for ztn-edge-router
     Loaded: loaded (/etc/systemd/system/ziti-router.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-12-10 09:37:29 UTC; 27min ago
   Main PID: 127525 (ziti)
      Tasks: 9 (limit: 6970)
     Memory: 33.9M
        CPU: 8.191s
     CGroup: /system.slice/ziti-router.service
             └─127525 /root/.ziti/quickstart/ztn/ziti-bin/ziti-v0.30.0/ziti router run /root/.ziti/quickstart/ztn/ztn-edge-router.yaml

Dec 10 10:03:57 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{PlWk}","chSeq":937,"connId":944,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:03:57.325Z","token":"122612ce-7f09-4ec8-a8b5-4ca39dabd8d7","type":"EdgeConnectType"}
Dec 10 10:03:57 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{PlWk}","chSeq":938,"connId":945,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:03:57.573Z","token":"122612ce-7f09-4ec8-a8b5-4ca39dabd8d7","type":"EdgeConnectType"}
Dec 10 10:03:57 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{PlWk}","chSeq":939,"connId":946,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:03:57.845Z","token":"122612ce-7f09-4ec8-a8b5-4ca39dabd8d7","type":"EdgeConnectType"}
Dec 10 10:03:58 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{PlWk}","chSeq":940,"connId":947,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:03:58.096Z","token":"122612ce-7f09-4ec8-a8b5-4ca39dabd8d7","type":"EdgeConnectType"}
Dec 10 10:03:58 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{PlWk}","chSeq":941,"connId":948,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:03:58.358Z","token":"122612ce-7f09-4ec8-a8b5-4ca39dabd8d7","type":"EdgeConnectType"}
Dec 10 10:05:18 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{5xnP}","chSeq":98,"connId":80,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:05:18.939Z","token":"9702a546-37fd-48b1-92d4-b66cc8e7c2fb","type":"EdgeConnectType"}
Dec 10 10:05:19 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{5xnP}","chSeq":99,"connId":81,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:05:19.188Z","token":"9702a546-37fd-48b1-92d4-b66cc8e7c2fb","type":"EdgeConnectType"}
Dec 10 10:05:19 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{5xnP}","chSeq":100,"connId":82,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:05:19.450Z","token":"9702a546-37fd-48b1-92d4-b66cc8e7c2fb","type":"EdgeConnectType"}
Dec 10 10:05:19 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{5xnP}","chSeq":101,"connId":83,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:05:19.701Z","token":"9702a546-37fd-48b1-92d4-b66cc8e7c2fb","type":"EdgeConnectType"}
Dec 10 10:05:19 ztn ziti[127525]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{5xnP}","chSeq":102,"connId":84,"edgeSeq":0,"error":"service 2j4hPkMjqSoYM6O4e4DVfp has no terminators","file":"github.com/openziti/edge@v0.24.381/router/xgress_edge/listener.go:171","func":"github.com/openziti/edge/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2023-12-10T10:05:19.968Z","token":"9702a546-37fd-48b1-92d4-b66cc8e7c2fb","type":"EdgeConnectType"}
~

Here are the other ziti-edge-tunnel.service 2 outputs (not the above issue or output)

Websever1 (working one. Ubuntu 20.04 LTS)

● ziti-edge-tunnel.service - Ziti Edge Tunnel
     Loaded: loaded (/etc/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-12-10 12:54:51 +03; 5min ago
    Process: 3698482 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
   Main PID: 3698495 (ziti-edge-tunne)
      Tasks: 5 (limit: 4631)
     Memory: 8.6M
     CGroup: /system.slice/ziti-edge-tunnel.service
             └─3698495 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

 10 12:58:36 odoo-ztn ziti-edge-tunnel[3698495]: (3698495)[      225.255]    INFO tunnel-cbs:ziti_hosting.c:443 on_hosted_client_connect() hosted_service[odoo-web], client[josh-pc] dst_addr[tcp:odooweb.ziti:8069]: incoming connection
 10 12:58:36 odoo-ztn ziti-edge-tunnel[3698495]: (3698495)[      225.258]    INFO tunnel-cbs:ziti_hosting.c:443 on_hosted_client_connect() hosted_service[odoo-web], client[josh-pc] dst_addr[tcp:odooweb.ziti:8069]: incoming connection
 10 12:58:36 odoo-ztn ziti-edge-tunnel[3698495]: (3698495)[      225.258]    INFO tunnel-cbs:ziti_hosting.c:443 on_hosted_client_connect() hosted_service[odoo-web], client[josh-pc] dst_addr[tcp:odooweb.ziti:8069]: incoming connection
 10 12:58:36 odoo-ztn ziti-edge-tunnel[3698495]: (3698495)[      225.261]    INFO tunnel-cbs:ziti_hosting.c:443 on_hosted_client_connect() hosted_service[odoo-web], client[josh-pc] dst_addr[tcp:odooweb.ziti:8069]: incoming connection
 10 12:58:36 odoo-ztn ziti-edge-tunnel[3698495]: (3698495)[      225.261]    INFO tunnel-cbs:ziti_hosting.c:443 on_hosted_client_connect() hosted_service[odoo-web], client[josh-pc] dst_addr[tcp:odooweb.ziti:8069]: incoming connection
 10 12:58:39 odoo-ztn ziti-edge-tunnel[3698495]: (3698495)[      228.547]    INFO tunnel-cbs:ziti_hosting.c:443 on_hosted_client_connect() hosted_service[odoo-web], client[josh-pc] dst_addr[tcp:odooweb.ziti:8069]: incoming connection

Websever2 (not working, even after restart it didn't work. Ubuntu 18.04 LTS)

● ziti-edge-tunnel.service - Ziti Edge Tunnel
   Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ziti-edge-tunnel.service.d
           └─10-run-as-root.conf
   Active: active (running) since Sun 2023-12-10 13:26:59 +03; 27s ago
  Process: 13543 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
 Main PID: 13546 (ziti-edge-tunne)
    Tasks: 5 (limit: 4657)
   CGroup: /system.slice/ziti-edge-tunnel.service
           └─13546 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

 10 13:26:59 odoo-prod systemd[1]: Starting Ziti Edge Tunnel...
 10 13:26:59 odoo-prod ziti-edge-tunnel.sh[13543]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
 10 13:26:59 odoo-prod systemd[1]: Started Ziti Edge Tunnel.
 10 13:26:59 odoo-prod ziti-edge-tunnel[13546]: (13546)[        0.038]    WARN ziti-edge-tunnel:resolvers.c:317 set_systemd_resolved_link_setting() Attempted to call unknown method: SetLinkDNSOverTLS for link: (ziti0)
 10 13:27:14 odoo-prod ziti-edge-tunnel[13546]: (13546)[       15.038]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[openziti.local] request failed: -110(connection timed out)
 10 13:27:14 odoo-prod ziti-edge-tunnel[13546]: (13546)[       15.038]   ERROR ziti-sdk:ziti.c:1515 version_cb() ztx[0] failed to get controller version from https://openziti.local:8441 CONTROLLER_UNAVAILABLE(connection timed out)
 10 13:27:14 odoo-prod ziti-edge-tunnel[13546]: (13546)[       15.038]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[openziti.local] request failed: -110(connection timed out)
 10 13:27:14 odoo-prod ziti-edge-tunnel[13546]: (13546)[       15.038]    WARN ziti-sdk:ziti.c:1444 api_session_cb() ztx[0] failed to get api session from ctrl[https://openziti.local:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] connection timed out
 10 13:27:14 odoo-prod ziti-edge-tunnel[13546]: (13546)[       15.038]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:744 on_ziti_event() ziti_ctx controller connections failed: ziti controller is not available
 10 13:27:14 odoo-prod ziti-edge-tunnel[13546]: (13546)[       15.038]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1199 on_event() ztx[/opt/openziti/etc/identities/odoo-prod-pc.json] failed to connect to controller due to ziti controller is not available

Thank you

scareything · December 10, 2023, 12:26pm

The ziti-edge-tunnel process on the webserver2 host cannot reach your controller. Things that I can think of to check:

Is webserver2 host on the same private network as the controller? If not, is there a route (as in ip route) that enables connections to the controller?
Does the "openziti.local" hostname resolve to the expected IP for the controller?
Are there any firewall rules on the controller (or the gateways / load balancers that connect it) that prevent connections on the controller ports from the webserver2 IP?

kashif · December 11, 2023, 10:04am

Yes on the same subnet.

default via 192.168.17.x dev ens160 proto static metric 100
x.137.234.x via 192.168.17.x dev ens160 metric 100
100.64.0.0/10 dev ziti0 scope link
100.64.0.2 dev ziti0 scope link
100.64.0.3 dev ziti0 scope link
169.254.0.0/16 dev ens160 scope link metric 1000
192.168.17.0/24 dev ens160 proto kernel scope link src 192.168.17.14 metric 100

Yes "openziti.local" ping responses back from Webserver2 once I have removed google DNS IP. It was there for sometime!

No.

Today I checked "erpaman.ziti:8069" and its working fine but sometimes with little delays and sometimes no response, had to refresh the page again.

This is the output for erpaman.

● ziti-edge-tunnel.service - Ziti Edge Tunnel
   Loaded: loaded (/lib/systemd/system/ziti-edge-tunnel.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ziti-edge-tunnel.service.d
           └─10-run-as-root.conf
   Active: active (running) since Mon 2023-12-11 12:50:37 +03; 35s ago
  Process: 10236 ExecStartPre=/opt/openziti/bin/ziti-edge-tunnel.sh (code=exited, status=0/SUCCESS)
 Main PID: 10237 (ziti-edge-tunne)
    Tasks: 9 (limit: 4915)
   CGroup: /system.slice/ziti-edge-tunnel.service
           └─10237 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities

12:50:37 odoo-prod systemd[1]: Starting Ziti Edge Tunnel...
12:50:37 odoo-prod systemd[1]: Started Ziti Edge Tunnel.
12:50:37 odoo-prod ziti-edge-tunnel[10237]: (10237)[        0.036]    WARN ziti-edge-tunnel:resolvers.c:317 set_systemd_resolved_link_setting() A
ttempted to call unknown method: SetLinkDNSOverTLS for link: (ziti0)
12:50:56 odoo-prod ziti-edge-tunnel[10237]: (10237)[       18.583]    WARN ziti-sdk:channel.c:337 on_channel_send() ch[0] write delay = 5.759 q=2
 qs=20284
13:14:47 odoo-prod ziti-edge-tunnel[10237]: (10237)[     1449.744]    WARN ziti-sdk:connect.c:1254 queue_edge_message() conn[0.50/Disconnected] disconnecting from state[6]
13:14:47 odoo-prod ziti-edge-tunnel[10237]: (10237)[     1449.744]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.49] closing bridge due to error: -21(connection to edge router terminated)
13:14:47 odoo-prod ziti-edge-tunnel[10237]: (10237)[     1449.744]    WARN ziti-sdk:conn_bridge.c:305 on_ziti_data() br[0.47] closing bridge due to error: -21(connection to edge router terminated)
13:14:47 odoo-prod ziti-edge-tunnel[10237]: (10237)[     1449.744]    WARN ziti-sdk:bind.c:346 on_message() binding failed: -21/connection to edge router terminated
13:14:52 odoo-prod ziti-edge-tunnel[10237]: (10237)[     1454.943]   ERROR ziti-sdk:channel.c:637 hello_reply_cb() ch[0] failed to receive Hello response due to -21(connection to edge router terminated)
13:15:46 odoo-prod ziti-edge-tunnel[10237]: (10237)[     1508.601]    WARN ziti-sdk:channel.c:337 on_channel_send() ch[0] write delay = 1.316 q=28 qs=356330

Thank you for your support.

scareything · December 11, 2023, 3:23pm

Hello,

I assume this is with ziti-edge-tunnel 0.22.16? It's worth trying with 0.22.17 which was released over the weekend.

kashif · December 12, 2023, 8:49am

Hi,
Thank you for your quick response. Much much appreciated
Yes, I have updated the Edge Tunneler to the latest (0.22.17) on all web servers.
It seems working fine now.

Thank you for your support.

Topic		Replies	Views
Service still can be accessed either IP or testpage.ziti	15	242	August 21, 2023
Intermittent Connectivity Issues: ziti-edge-tunnel Support	4	25	November 21, 2024
Service fails in Ziti Edge Tunnels Gw setup Ziti Edge Tunnel	15	241	February 29, 2024
Ziti edge client is running but not showing on ZAC	6	230	August 24, 2023
OpenZiti performance problem for help	11	516	April 14, 2023

Slow response and sometimes website inaccessible

Related topics