Service fails in Ziti Edge Tunnels Gw setup

Tunneler Apps Ziti Edge Tunnel
1

We're attempting to replicate the scenario outlined in the link Ziti-Edge-Tunnel as Gateway | OpenZiti with a slight modification. In our setup, we have two Ziti Edge Tunnels (local-tunnel and remote-tunnel), and a single Ubuntu 22.04 server (Ubuntu Server) located in the same subnet as the remote-tunnel. The scenario is shown bellow

ziti2
ziti21371×792 41 KB

On the Ubuntu server, we've launched a simple Python server running on port 8080. Our goal is to use curl from the OpenZiti Edge Tunnel gateway (local-tunnel with IP 10.10.6.13) to access this Python server running on Ubuntu server at IP 192.168.5.220. Strangely, we're only successful when we configure the intercept.v1 configuration with a DNS name (mysimpleservice.ziti). The configuration of services and policies have been done as described in the guide. However, if we set as “addresses” in the intercept.v1 configuration the IP address of the Ubuntu server (192.168.5.220), we encounter a "connection refused" error. Another strange behavior is that before setting up the ziti service, 192.168.5.56 can curl the 192.168.5.220:8080. However after setting up the ziti service, 192.168.5.56 also receives a connection refused.

We've deployed the OpenZiti Edge Tunnel gateways both as native services and also as Docker containers, using the openziti/ziti-edge-tunnel:0.22.15 Docker image. For the OpenZiti Controller, we've deployed version 0.31.4 following the Express Install guide.

Below are the logs from the OpenZiti Edge Tunnel (local-tunnel), as well as the output of the OpenZiti Controller's services ziti-route and ziti-controller.

systemctl status ziti-router -l --no-pager
● ziti-router.service - Ziti-Router for ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router
     Loaded: loaded (/etc/systemd/system/ziti-router.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-02-26 14:19:33 UTC; 1h 30min ago
   Main PID: 194385 (ziti)
      Tasks: 11 (limit: 19179)
     Memory: 72.6M
        CPU: 1min 20.724s
     CGroup: /system.slice/ziti-router.service
             └─194385 /root/.ziti/quickstart/ziti-instance-1.europe-west6-a.c.ziticontroller.internal/ziti-bin/ziti-v0.32.2/ziti router run /root/.ziti/quickstart/ziti-instance-1.europe-west6-a.c.ziticontroller.internal/ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router.yaml

Feb 26 15:48:43 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: [635B blob data]
Feb 26 15:48:43 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:253","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminatorWithRetry","level":"info","msg":"terminator validated successfully","terminatorId":"4cmI9viSwvWue6R0Gbmu0f","time":"2024-02-26T15:48:43.726Z"}
Feb 26 15:48:43 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:233","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminatorWithRetry.func1","level":"info","msg":"attempting to establish terminator","terminatorId":"1u8YvoRMhZwNsJfvbFl3Ov","time":"2024-02-26T15:48:43.726Z"}
Feb 26 15:48:43 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:318","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).HandleCreateTerminatorResponse","level":"info","msg":"received terminator created notification","routerId":"KaX7V9DzsY","terminatorId":"1u8YvoRMhZwNsJfvbFl3Ov","time":"2024-02-26T15:48:43.731Z"}
Feb 26 15:48:43 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"file":"github.com/openziti/ziti/router/xgress_edge/hosted.go:253","func":"github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).establishTerminatorWithRetry","level":"info","msg":"terminator validated successfully","terminatorId":"1u8YvoRMhZwNsJfvbFl3Ov","time":"2024-02-26T15:48:43.772Z"}
Feb 26 15:49:44 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"_channels":["establishPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attempt":0,"attemptNumber":"1","binding":"edge","circuitId":"TV4zS.dNe","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{G0bO}","destination":"1u8YvoRMhZwNsJfvbFl3Ov","error":"error creating route for [c/TV4zS.dNe]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/ziti/router/handler_ctrl/route.go:138","func":"github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failure while handling route update","serviceId":"3mwWanRJFEphQlXnA18Tlu","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:49:44.797Z"}
Feb 26 15:49:49 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"_channels":["establishPath"],"apiSessionId":"clt345u0q02o5jbakzohyceoc","attempt":0,"attemptNumber":"1","binding":"edge","circuitId":"9W4zBLdNe","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{G0bO}","destination":"1u8YvoRMhZwNsJfvbFl3Ov","error":"error creating route for [c/9W4zBLdNe]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/ziti/router/handler_ctrl/route.go:138","func":"github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failure while handling route update","serviceId":"3mwWanRJFEphQlXnA18Tlu","sessionId":"clt34721p02pwjbakuqbnwgu8","time":"2024-02-26T15:49:49.798Z"}
Feb 26 15:49:54 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"_channels":["establishPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attempt":1,"attemptNumber":"2","binding":"edge","circuitId":"TV4zS.dNe","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{G0bO}","destination":"KO4wecoq1ilmVtjYYhAWK","error":"error creating route for [c/TV4zS.dNe]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/ziti/router/handler_ctrl/route.go:138","func":"github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failure while handling route update","serviceId":"3mwWanRJFEphQlXnA18Tlu","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:49:54.798Z"}
Feb 26 15:49:54 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"_channels":["establishPath"],"apiSessionId":"clt345u0q02o5jbakzohyceoc","attempt":0,"attemptNumber":"1","binding":"edge","circuitId":"uudzB.lvo","context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{G0bO}","destination":"KO4wecoq1ilmVtjYYhAWK","error":"error creating route for [c/uudzB.lvo]: timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/ziti/router/handler_ctrl/route.go:138","func":"github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail","level":"error","msg":"failure while handling route update","serviceId":"3mwWanRJFEphQlXnA18Tlu","sessionId":"clt34721p02pwjbakuqbnwgu8","time":"2024-02-26T15:49:54.799Z"}
Feb 26 15:49:54 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[194385]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{56WY}","chSeq":10563,"connId":10445,"edgeSeq":0,"error":"exceeded maximum [2] retries creating circuit [c/TV4zS.dNe]: timeout creating routes for [s/TV4zS.dNe]","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:175","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-02-26T15:49:54.809Z","token":"87e2b75f-1a1a-49b1-acb8-59aa91e3911c","type":"EdgeConnectType"}

systemctl status ziti-controller -l --no-pager
● ziti-controller.service - Ziti-Controller
     Loaded: loaded (/etc/systemd/system/ziti-controller.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-02-26 14:51:35 UTC; 58min ago
   Main PID: 195095 (ziti)
      Tasks: 11 (limit: 19179)
     Memory: 115.8M
        CPU: 45.675s
     CGroup: /system.slice/ziti-controller.service
             └─195095 /root/.ziti/quickstart/ziti-instance-1.europe-west6-a.c.ziticontroller.internal/ziti-bin/ziti-v0.32.2/ziti controller run /root/.ziti/quickstart/ziti-instance-1.europe-west6-a.c.ziticontroller.internal/ziti-instance-1.europe-west6-a.c.ziticontroller.internal.yaml

Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_channels":["selectPath"],"apiSessionId":"clt345u0q02o5jbakzohyceoc","attemptNumber":3,"circuitId":"uudzB.lvo","file":"github.com/openziti/ziti/controller/network/network.go:562","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"circuit creation failed after [2] attempts, sending cleanup unroutes","serviceId":"3mwWanRJFEphQlXnA18Tlu","serviceName":"svc2","sessionId":"clt34721p02pwjbakuqbnwgu8","time":"2024-02-26T15:50:04.803Z"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_context":"ch{KaX7V9DzsY}-\u003eu{classic}-\u003ei{G0bO}","error":"exceeded maximum [2] retries creating circuit [c/uudzB.lvo]: error creating route for [s/uudzB.lvo] on [r/KaX7V9DzsY] (error creating route for [c/uudzB.lvo]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"KaX7V9DzsY","time":"2024-02-26T15:50:04.803Z","token":"ce2e50b9-f9c7-4d45-8b24-d5dfc0195cb0"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_channels":["establishPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attemptNumber":2,"circuitId":"ytyMBLlNo","file":"github.com/openziti/ziti/controller/network/routesender.go:196","func":"github.com/openziti/ziti/controller/network.(*routeSender).handleRouteSend","level":"warning","msg":"received failed route status from [r/KaX7V9DzsY] for attempt [#1] of [s/ytyMBLlNo] (error creating route for [c/ytyMBLlNo]: timeout waiting for message reply: context deadline exceeded)","serviceId":"3mwWanRJFEphQlXnA18Tlu","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:50:04.803Z"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_channels":["selectPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attemptNumber":2,"circuitId":"ytyMBLlNo","error":"error creating route for [s/ytyMBLlNo] on [r/KaX7V9DzsY] (error creating route for [c/ytyMBLlNo]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/ziti/controller/network/network.go:553","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","serviceId":"3mwWanRJFEphQlXnA18Tlu","serviceName":"svc2","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:50:04.803Z"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_channels":["selectPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attemptNumber":3,"circuitId":"ytyMBLlNo","file":"github.com/openziti/ziti/controller/network/network.go:562","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"circuit creation failed after [2] attempts, sending cleanup unroutes","serviceId":"3mwWanRJFEphQlXnA18Tlu","serviceName":"svc2","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:50:04.803Z"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_context":"ch{KaX7V9DzsY}-\u003eu{classic}-\u003ei{G0bO}","error":"exceeded maximum [2] retries creating circuit [c/ytyMBLlNo]: error creating route for [s/ytyMBLlNo] on [r/KaX7V9DzsY] (error creating route for [c/ytyMBLlNo]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"KaX7V9DzsY","time":"2024-02-26T15:50:04.803Z","token":"87e2b75f-1a1a-49b1-acb8-59aa91e3911c"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_channels":["establishPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attemptNumber":2,"circuitId":"zayMSLlvo","file":"github.com/openziti/ziti/controller/network/routesender.go:196","func":"github.com/openziti/ziti/controller/network.(*routeSender).handleRouteSend","level":"warning","msg":"received failed route status from [r/KaX7V9DzsY] for attempt [#1] of [s/zayMSLlvo] (error creating route for [c/zayMSLlvo]: timeout waiting for message reply: context deadline exceeded)","serviceId":"3mwWanRJFEphQlXnA18Tlu","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:50:04.803Z"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_channels":["selectPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attemptNumber":2,"circuitId":"zayMSLlvo","error":"error creating route for [s/zayMSLlvo] on [r/KaX7V9DzsY] (error creating route for [c/zayMSLlvo]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/ziti/controller/network/network.go:553","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"route attempt for circuit failed","serviceId":"3mwWanRJFEphQlXnA18Tlu","serviceName":"svc2","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:50:04.804Z"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_channels":["selectPath"],"apiSessionId":"clt33ijx901tkjbak7rplusiq","attemptNumber":3,"circuitId":"zayMSLlvo","file":"github.com/openziti/ziti/controller/network/network.go:562","func":"github.com/openziti/ziti/controller/network.(*Network).CreateCircuit","level":"warning","msg":"circuit creation failed after [2] attempts, sending cleanup unroutes","serviceId":"3mwWanRJFEphQlXnA18Tlu","serviceName":"svc2","sessionId":"clt3471up02ptjbak08dixxol","time":"2024-02-26T15:50:04.804Z"}
Feb 26 15:50:04 ziti-instance-1.europe-west6-a.c.ziticontroller.internal ziti[195095]: {"_context":"ch{KaX7V9DzsY}-\u003eu{classic}-\u003ei{G0bO}","error":"exceeded maximum [2] retries creating circuit [c/zayMSLlvo]: error creating route for [s/zayMSLlvo] on [r/KaX7V9DzsY] (error creating route for [c/zayMSLlvo]: timeout waiting for message reply: context deadline exceeded)","file":"github.com/openziti/ziti/controller/handler_edge_ctrl/common.go:75","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*baseRequestHandler).returnError","level":"error","msg":"responded with error","operation":"create.circuit","routerId":"KaX7V9DzsY","time":"2024-02-26T15:50:04.804Z","token":"87e2b75f-1a1a-49b1-acb8-59aa91e3911c"}

The logs from OpenZiti Edge Tunnel (local-tunnel) deployed as docker

(68560)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
(68560)[        0.000]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.36.6 @9cda05d(HEAD) starting at (2024-02-26T15:48:42.691)
(68560)[        0.000]    INFO ziti-edge-tunnel:instance-config.c:86 load_tunnel_status_from_file() Loading config file from /var/lib/ziti/config.json
(68560)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
(68560)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.23)
(68560)[        0.000]    INFO tunnel-cbs:ziti_dns.c:170 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(68560)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1698 run_tunneler_loop() Loading identity files from /opt/openziti/etc/identities
(68560)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1624 make_socket_path() effective group set to 'ziti' (gid=998)
(68560)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1102 load_identities() loading identity file: prx56.json
(68560)[        0.000]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/opt/openziti/etc/identities/prx56.json] is not loaded yet or already removed.
(68560)[        0.010]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(68560)[        0.010]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/opt/openziti/etc/identities/prx56.json]
(68560)[        0.010]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/opt/openziti/etc/identities/prx56.json]
(68560)[        0.010]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1117 load_id_cb() identity[/opt/openziti/etc/identities/prx56.json] loaded
(68560)[        0.011]    INFO ziti-sdk:ziti.c:450 ziti_init_async() ztx[0] using tlsuv[v0.28.2], tls[Mbed TLS 2.28.5]
(68560)[        0.011]    INFO ziti-sdk:ziti.c:451 ziti_init_async() ztx[0] Loading ziti context with controller[https://ztc.euprojects.net:8441]
(68560)[        0.011]    INFO ziti-sdk:ziti.c:929 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://ztc.euprojects.net:8441] api_session_status[0] api_session_expired[TRUE]
(68560)[        0.012]    INFO ziti-edge-tunnel:resolvers.c:356 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(68560)[        0.218]    INFO ziti-sdk:ziti.c:1693 version_cb() ztx[0] connected to controller https://ztc.euprojects.net:8441 version v0.31.4(1c21434737ac 2023-12-18T16:24:57Z)
(68560)[        0.269]    INFO ziti-sdk:ziti.c:1583 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(68560)[        0.269]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[prx56] connected to controller
(68560)[        0.269]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1147 on_event() ztx[/opt/openziti/etc/identities/prx56.json] context event : status is OK
(68560)[        0.459]    INFO ziti-sdk:channel.c:271 new_ziti_channel() ch[0] (ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router) new channel for ztx[0] identity[prx56]
(68560)[        0.459]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[prx56] added edge router ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router@ztc.euprojects.net
(68560)[        0.459]    INFO ziti-sdk:channel.c:777 reconnect_channel() ch[0] reconnecting NOW
(68560)[        0.557]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[svc] with intercept.v1 = {"addresses":["192.168.5.56"],"portRanges":[{"high":8080,"low":8080}],"protocols":["tcp"]}
(68560)[        0.557]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[svc]
(68560)[        0.557]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:192.168.5.56:8080] service[svc]
(68560)[        0.557]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[svc2] with intercept.v1 = {"addresses":["192.168.5.220"],"portRanges":[{"high":8080,"low":8080}],"protocols":["tcp"]}
(68560)[        0.557]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[svc2]
(68560)[        0.557]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:192.168.5.220:8080] service[svc2]
(68560)[        0.557]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - svc:4zH6B381sHxOjpr92w9MkB ===============
(68560)[        0.557]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - svc2:3mwWanRJFEphQlXnA18Tlu ===============
(68560)[        0.557]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 3 route updates
(68560)[        0.559]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[3]: 0/OK
(68560)[        0.730]    INFO ziti-sdk:channel.c:669 hello_reply_cb() ch[0] connected. EdgeRouter version: v0.31.4|1c21434737ac|2023-12-18T16:24:57Z|linux|amd64
(68560)[        0.730]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:843 on_ziti_event() ztx[prx56] router ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router connected
(68560)[        1.269]    INFO ziti-sdk:posture.c:202 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(68560)[       57.124]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:38982] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       62.126]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:49644] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       77.128]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:57198] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       77.129]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51400] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       77.129]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51394] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       77.129]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51410] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       77.130]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.3/jQlGfsuL/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/9W4zBLdNe]: timeout creating routes for [s/9W4zBLdNe]
(68560)[       77.130]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       77.130]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:54556] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       77.130]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:54546] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       77.130]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[10.10.6.13] client_src_addr[tcp:100.64.0.1:38982]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       82.129]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:41802] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.129]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:41830] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.129]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:41810] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.129]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:41808] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.131]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:41820] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.131]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[prx56] client_src_addr[tcp:100.64.0.1:41836] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.132]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51394] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.132]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51410] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.133]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.5/2qFpdS7K/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/uudzB.lvo]: error creating route for [s/uudzB.lvo] on [r/KaX7V9DzsY] (error creating route for [c/uudzB.lvo]: timeout waiting for message reply: context deadline exceeded)
(68560)[       82.133]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       82.133]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51400] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       82.133]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[prx56] client_src_addr[tcp:100.64.0.1:49644]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       92.135]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.9/GWilrVkp/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/annzB.lNor]: error creating route for [s/annzB.lNor] on [r/KaX7V9DzsY] (error creating route for [c/annzB.lNor]: timeout waiting for message reply: context deadline exceeded)
(68560)[       92.135]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       92.135]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.8/z-qoDxTp/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/FniMSLdNo]: error creating route for [s/FniMSLdNo] on [r/KaX7V9DzsY] (error creating route for [c/FniMSLdNo]: timeout waiting for message reply: context deadline exceeded)
(68560)[       92.135]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       92.135]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51400]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       92.135]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[prx56] client_src_addr[tcp:100.64.0.1:57198]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       92.137]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.17/BsmHJr5_/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/UZizSLlNo]: error creating route for [s/UZizSLlNo] on [r/KaX7V9DzsY] (error creating route for [c/UZizSLlNo]: timeout waiting for message reply: context deadline exceeded)
(68560)[       92.137]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       92.137]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[10.10.6.13] client_src_addr[tcp:100.64.0.1:54546]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       92.138]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.16/Yfg3KHy0/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/AiiMS.lNe]: error creating route for [s/AiiMS.lNe] on [r/KaX7V9DzsY] (error creating route for [c/AiiMS.lNe]: timeout waiting for message reply: context deadline exceeded)
(68560)[       92.138]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       92.138]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.12/4gas-fjG/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/7nnMBLlve]: error creating route for [s/7nnMBLlve] on [r/KaX7V9DzsY] (error creating route for [c/7nnMBLlve]: timeout waiting for message reply: context deadline exceeded)
(68560)[       92.138]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       92.138]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[10.10.6.13] client_src_addr[tcp:100.64.0.1:54556]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       92.138]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51394]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       92.140]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.13/4B2zeCSo/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/aniMBLlNop]: error creating route for [s/aniMBLlNop] on [r/KaX7V9DzsY] (error creating route for [c/aniMBLlNop]: timeout waiting for message reply: context deadline exceeded)
(68560)[       92.140]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
(68560)[       92.140]   ERROR tunnel-cbs:ziti_hosting.c:222 on_hosted_tcp_server_connect_complete() hosted_service[svc2], client[10.10.6.13] client_src_addr[tcp:100.64.0.1:51410]: connect to tcp:192.168.5.220:8080 failed: connection refused
(68560)[       92.179]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:54724] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(68560)[       92.180]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:54758] dst_addr[tcp:192.168.5.220:8080]: incoming connection

We'd appreciate any insights or guidance on resolving this issue.

2 Likes
2

The log is showing a connection refused to tcp:192.168.5.220:8080. Is the configuration at this time different than your diagram? The service should be forwarding to .77, correct?

Can you post the output of the services and configs? Then we can determine if it is configured correctly.

ziti edge list services -j
ziti edge list service-congis -j

3

You are correct, there is a misunderasting, the Ubuntu server is on 192.168.5.220 IP and also the python http server running on port 8080. I will update also the image on the original post. In addition , the output for services, policies and configs

Services

 ziti edge list services -j
{
    "data": [
        {
            "_links": {
                "configs": {
                    "href": "./services/744C670PSrilutQ5KcoATW/configs"
                },
                "self": {
                    "href": "./services/744C670PSrilutQ5KcoATW"
                },
                "service-edge-router-policies": {
                    "href": "./services/744C670PSrilutQ5KcoATW/service-edge-router-policies"
                },
                "service-policies": {
                    "href": "./services/744C670PSrilutQ5KcoATW/service-policies"
                },
                "terminators": {
                    "href": "./services/744C670PSrilutQ5KcoATW/terminators"
                }
            },
            "createdAt": "2024-02-27T11:52:05.339Z",
            "id": "744C670PSrilutQ5KcoATW",
            "tags": {},
            "updatedAt": "2024-02-27T12:13:10.931Z",
            "config": {},
            "configs": [
                "1zfsfhe0zfTWB0heapmMKI",
                "7cuQH7Wo9QvTFlJLbF2jRL"
            ],
            "encryptionRequired": true,
            "maxIdleTimeMillis": 0,
            "name": "svc",
            "permissions": [
                "Bind",
                "Dial"
            ],
            "postureQueries": [],
            "roleAttributes": null,
            "terminatorStrategy": "smartrouting"
        }
    ],
    "meta": {
        "filterableFields": [
            "isSystem",
            "terminatorStrategy",
            "tags",
            "roleAttributes",
            "id",
            "updatedAt",
            "configs",
            "createdAt",
            "name"
        ],
        "pagination": {
            "limit": 10,
            "offset": 0,
            "totalCount": 1
        }
    }
}

Configs

ziti edge list configs -j
{
    "data": [
        {
            "_links": {
                "self": {
                    "href": "./configs/1zfsfhe0zfTWB0heapmMKI"
                }
            },
            "createdAt": "2024-02-26T15:11:23.533Z",
            "id": "1zfsfhe0zfTWB0heapmMKI",
            "tags": {},
            "updatedAt": "2024-02-27T11:38:33.104Z",
            "configType": {
                "_links": {
                    "self": {
                        "href": "./config-types/NH5p4FpGR"
                    }
                },
                "entity": "config-types",
                "id": "NH5p4FpGR",
                "name": "host.v1"
            },
            "configTypeId": "NH5p4FpGR",
            "data": {
                "address": "192.168.5.220",
                "listenOptions": {
                    "bindUsingEdgeIdentity": false,
                    "precedence": "default"
                },
                "port": 8080,
                "protocol": "tcp"
            },
            "name": "svc2-Host"
        },
        {
            "_links": {
                "self": {
                    "href": "./configs/7cuQH7Wo9QvTFlJLbF2jRL"
                }
            },
            "createdAt": "2024-02-26T15:02:02.855Z",
            "id": "7cuQH7Wo9QvTFlJLbF2jRL",
            "tags": {},
            "updatedAt": "2024-02-27T11:47:02.650Z",
            "configType": {
                "_links": {
                    "self": {
                        "href": "./config-types/g7cIWbcGg"
                    }
                },
                "entity": "config-types",
                "id": "g7cIWbcGg",
                "name": "intercept.v1"
            },
            "configTypeId": "g7cIWbcGg",
            "data": {
                "addresses": [
                    "192.168.5.220"
                ],
                "portRanges": [
                    {
                        "high": 80,
                        "low": 80
                    }
                ],
                "protocols": [
                    "tcp"
                ]
            },
            "name": "svc2-Client"
        }
    ],
    "meta": {
        "filterableFields": [
            "updatedAt",
            "tags",
            "isSystem",
            "name",
            "type",
            "id",
            "createdAt"
        ],
        "pagination": {
            "limit": 10,
            "offset": 0,
            "totalCount": 2
        }
    }
}

Service Policies

ziti edge list service-policies -j
{
    "data": [
        {
            "_links": {
                "identities": {
                    "href": "./service-policies/3LMH6bNLzJsofUospxoLzf/identities"
                },
                "posture-checks": {
                    "href": "./service-policies/3LMH6bNLzJsofUospxoLzf/posture-checks"
                },
                "self": {
                    "href": "./service-policies/3LMH6bNLzJsofUospxoLzf"
                },
                "services": {
                    "href": "./service-policies/3LMH6bNLzJsofUospxoLzf/services"
                }
            },
            "createdAt": "2024-02-27T11:53:43.346Z",
            "id": "3LMH6bNLzJsofUospxoLzf",
            "tags": {},
            "updatedAt": "2024-02-27T11:53:43.346Z",
            "identityRoles": [
                "@tDO3Y0vVn"
            ],
            "identityRolesDisplay": [
                {
                    "name": "@prx56",
                    "role": "@tDO3Y0vVn"
                }
            ],
            "name": "svc-Bind",
            "postureCheckRoles": null,
            "postureCheckRolesDisplay": [],
            "semantic": "AnyOf",
            "serviceRoles": [
                "@744C670PSrilutQ5KcoATW"
            ],
            "serviceRolesDisplay": [
                {
                    "name": "@svc",
                    "role": "@744C670PSrilutQ5KcoATW"
                }
            ],
            "type": "Bind"
        },
        {
            "_links": {
                "identities": {
                    "href": "./service-policies/6liwzXYGGLZD2pfm4p29WE/identities"
                },
                "posture-checks": {
                    "href": "./service-policies/6liwzXYGGLZD2pfm4p29WE/posture-checks"
                },
                "self": {
                    "href": "./service-policies/6liwzXYGGLZD2pfm4p29WE"
                },
                "services": {
                    "href": "./service-policies/6liwzXYGGLZD2pfm4p29WE/services"
                }
            },
            "createdAt": "2024-02-27T11:53:29.492Z",
            "id": "6liwzXYGGLZD2pfm4p29WE",
            "tags": {},
            "updatedAt": "2024-02-27T12:13:02.558Z",
            "identityRoles": [
                "@C2Avb0vVO"
            ],
            "identityRolesDisplay": [
                {
                    "name": "@10.10.6.13",
                    "role": "@C2Avb0vVO"
                }
            ],
            "name": "svc-Dial",
            "postureCheckRoles": null,
            "postureCheckRolesDisplay": [],
            "semantic": "AnyOf",
            "serviceRoles": [
                "@744C670PSrilutQ5KcoATW"
            ],
            "serviceRolesDisplay": [
                {
                    "name": "@svc",
                    "role": "@744C670PSrilutQ5KcoATW"
                }
            ],
            "type": "Dial"
        }
    ],
    "meta": {
        "filterableFields": [
            "tags",
            "name",
            "identityRoles",
            "serviceRoles",
            "id",
            "createdAt",
            "type",
            "semantic",
            "postureCheckRoles",
            "updatedAt",
            "isSystem"
        ],
        "pagination": {
            "limit": 10,
            "offset": 0,
            "totalCount": 2
        }
    }
}

NOTE: We found the error. In the OpenZiti edge tunnel gw (remote-tunnel) the routing table is the following :
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.5.254 0.0.0.0 UG 100 0 0 eth0
34.65.229.131 192.168.5.254 255.255.255.255 UGH 100 0 0 eth0
100.64.0.0 0.0.0.0 255.192.0.0 U 0 0 0 ziti0
100.64.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 ziti0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.5.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
192.168.5.2 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
192.168.5.220 0.0.0.0 255.255.255.255 UH 0 0 0 ziti0
192.168.5.254 0.0.0.0 255.255.255.255 UH 100 0 0 eth0

As we can see there is a route entry

192.168.5.220   0.0.0.0         255.255.255.255 UH    0      0        0 ziti0

which sends the traffic for the 192.168.5.220 not through eth0 but to ziti0.

If we delete this route we can successfully curl 192.168.5.220:80 from the OpenZiti edge tunnel gw (local-tunnel) with IP 10.10.6.13. The problem is that this rule will be added again when we will restart the OpenZiti edge tunnel gw (remote-tunnel) (service or docker).

Any idea on this?

4

It appears that you configured the hosting node to also intercept, or perhaps you did at some time and the route wasn't cleaned up properly? The service configs look correct, one dial, one bind, different identities. You can always check that with the policy-advisor.

ziti edge policy-advisor services

Look for an identity that has both dial and bind permissions. That is almost always a bad thing (unless using addressable terminators) as it tends to form a loop where hosted traffic is emitted, ingested and round and round you go.

With the configs as you have them now, you should restart everything and make sure it works. If there is a problem, it should resurface, if the problem was caused during testing, it should not.

5

You said it would come back if you restarted, did you test that, or are you assuming?

6

Yes we tested , every time we restart the docker container of Openziti edge tunnel this route is added.

Also this is the ouput of ziti edge policy-advisor services

Policy General Guidelines
  In order for an identity to dial or bind a service, the following must be true:
    - The identity must have access to the service via a service policy of the correct type (dial or bind)
    - The identity must have acces to at least one on-line edge router via an edge router policy
    - The service must have access to at least one on-line edge router via a service edge router policy
    - There must be at least one on-line edge router that both the identity and service have access to.

Policy Advisor Output Guide:
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR.
  ID = identity name
  ID ROUTERS = number of routers accessible to the identity via edge router policies.
    - See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
  SVC = service name
  SVC ROUTERS = number of routers accessible to the service via service edge router policies.
    - See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
  ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
  COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
  DIAL_OK = indicates if the identity has permission to dial the service.
    - See service polices for a service  : ziti edge controller list service service-policies <service>
    - See service polices for an identity: ziti edge controller list identity service-policies <identity>
  BIND_OK = indicates if the identity has permission to bind the service.
  ERROR_LIST = if the status is ERROR, error details will be listed on the following lines

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
OKAY : 10.10.6.13 (1) -> svc2 (1) Common Routers: (1/1) Dial: Y Bind: N

OKAY : proxmox56 (1) -> svc2 (1) Common Routers: (1/1) Dial: N Bind: Y

OKAY : 10.10.6.13 (1) -> svc (1) Common Routers: (1/1) Dial: Y Bind: N

OKAY : proxmox56 (1) -> svc (1) Common Routers: (1/1) Dial: N Bind: Y
7

Hi,

Can you please share how you are launching the docker container (environment variables, etc), as well as the logs from the remote-tunnel (including when the process starts)?

8

Hi we are launching the docker container through the following docker-compose.yaml

version: "3.9"
services:
    ziti-tun:
        image: openziti/ziti-edge-tunnel:latest
        devices:
            - /dev/net/tun:/dev/net/tun
        volumes:
            - .:/ziti-edge-tunnel
            - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
        environment:
            - ZITI_IDENTITY_BASENAME=ziti_id
            - PFXLOG_NO_JSON=true              # suppress JSON logging
        network_mode: host
        privileged: true

Also the logs when starting thr remote-tunnel conatiner and execute from the other side (10.10.6.13) the command curl 192.168.5.220:80

docker logs ziti-ziti-tun-1 -f
DEBUG: waiting 3s for /ziti-edge-tunnel/ziti_id.json (or token) to appear
DEBUG: identity file /ziti-edge-tunnel/ziti_id.json not found
DEBUG: /var/run/secrets/netfoundry.io/enrollment-token/ziti_id.jwt not found
DEBUG: /enrollment-token/ziti_id.jwt not found
INFO: enrolling /ziti-edge-tunnel/ziti_id.jwt
(7)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
(7)[        0.000]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.36.5 @4271e3e(HEAD) starting at (2024-02-27T16:11:55.409)
(7)[        0.000]    INFO ziti-sdk:ziti_enroll.c:90 ziti_enroll() Ziti C SDK version 0.36.5 @4271e3e(HEAD) starting enrollment at (2024-02-27T16:11:55.409)
INFO: found identity file /ziti-edge-tunnel/ziti_id.json
DEBUG: checking for run mode as first positional in: run
INFO: running: ziti-edge-tunnel run --identity /ziti-edge-tunnel/ziti_id.json
DEBUG: waiting for ziti-edge-tunnel PID: 14
(14)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
(14)[        0.000]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.36.5 @4271e3e(HEAD) starting at (2024-02-27T16:11:56.869)
(14)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.22)
(14)[        0.000]    INFO tunnel-cbs:ziti_dns.c:170 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(14)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1624 make_socket_path() effective group set to 'ziti' (gid=2171)
(14)[        0.014]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(14)[        0.014]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/ziti-edge-tunnel/ziti_id.json]
(14)[        0.014]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/ziti-edge-tunnel/ziti_id.json]
(14)[        0.014]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1117 load_id_cb() identity[/ziti-edge-tunnel/ziti_id.json] loaded
(14)[        0.014]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/ziti-edge-tunnel/ziti_id.json] is not loaded yet or already removed.
(14)[        0.014]   ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
(14)[        0.014]   ERROR ziti-edge-tunnel:instance-config.c:142 save_tunnel_status_to_file() Could not open config file /var/lib/ziti/config.json to store the tunnel status data
(14)[        0.015]    INFO ziti-sdk:ziti.c:450 ziti_init_async() ztx[0] using tlsuv[v0.28.2], tls[Mbed TLS 2.28.5]
(14)[        0.015]    INFO ziti-sdk:ziti.c:451 ziti_init_async() ztx[0] Loading ziti context with controller[https://ztc.euprojects.net:8441]
(14)[        0.015]    INFO ziti-sdk:ziti.c:928 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://ztc.euprojects.net:8441] api_session_status[0] api_session_expired[TRUE]
(14)[        0.015]    INFO ziti-edge-tunnel:resolvers.c:356 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(14)[        0.220]    INFO ziti-sdk:ziti.c:1692 version_cb() ztx[0] connected to controller https://ztc.euprojects.net:8441 version v0.30.5(4f324bd22875 2023-10-13T20:22:56Z)
(14)[        0.270]    INFO ziti-sdk:ziti.c:1582 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(14)[        0.270]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[proxmox56] connected to controller
(14)[        0.270]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1147 on_event() ztx[/ziti-edge-tunnel/ziti_id.json] context event : status is OK
(14)[        0.464]    INFO ziti-sdk:channel.c:271 new_ziti_channel() ch[0] (ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router) new channel for ztx[0] identity[proxmox56]
(14)[        0.464]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[proxmox56] added edge router ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router@ztc.euprojects.net
(14)[        0.464]    INFO ziti-sdk:channel.c:777 reconnect_channel() ch[0] reconnecting NOW
(14)[        0.567]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[svc] with intercept.v1 = {"addresses":["192.168.5.56"],"portRanges":[{"high":8080,"low":8080}],"protocols":["tcp"]}
(14)[        0.567]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[svc]
(14)[        0.567]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:192.168.5.56:8080] service[svc]
(14)[        0.567]    INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[svc2] with intercept.v1 = {"addresses":["192.168.5.220"],"portRanges":[{"high":80,"low":80}],"protocols":["tcp"]}
(14)[        0.567]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[svc2]
(14)[        0.567]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:192.168.5.220:8080] service[svc2]
(14)[        0.567]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - svc:6Jg6PRKaS9Jkn81Ex2nhvg ===============
(14)[        0.567]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1272 on_event() =============== service event (added) - svc2:47VNysadrkoZ5rO74zfH9G ===============
(14)[        0.567]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 3 route updates
(14)[        0.569]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[3]: 0/OK
(14)[        0.764]    INFO ziti-sdk:channel.c:669 hello_reply_cb() ch[0] connected. EdgeRouter version: v0.30.5|4f324bd22875|2023-10-13T20:22:56Z|linux|amd64
(14)[        0.764]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:843 on_ziti_event() ztx[proxmox56] router ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router connected
(14)[        1.270]    INFO ziti-sdk:posture.c:202 ziti_send_posture_data() ztx[0] first run or potential controller restart detected



(14)[      386.936]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:48114] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(14)[      402.133]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[10.10.6.13] client_src_addr[tcp:100.64.0.1:37660] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(14)[      485.492]    INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[proxmox56] client_src_addr[tcp:100.64.0.1:37722] dst_addr[tcp:192.168.5.220:8080]: incoming connection
(14)[      490.494]   ERROR ziti-sdk:connect.c:965 connect_reply_cb() conn[0.4/IhAllyct/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/FuiNWbkPO]: error creating route for [s/FuiNWbkPO] on [r/8hMVTEY9MG] (error creating route for [c/FuiNWbkPO]: timeout waiting for message reply: context deadline exceeded)
(14)[      490.494]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
9

I just looked more closely at the logs you shared from the local-tunnel, and I see a problem there. You have the situation that @gormami described where a tunneler is intercepting and hosting the same IPs (once for service "svc" and again for service "svc2"):

INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[svc] with intercept.v1 = {"addresses":["192.168.5.56"],"portRanges":[{"high":8080,"low":8080}],"protocols":["tcp"]}
INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[svc]
INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:192.168.5.56:8080] service[svc]
INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[svc2] with intercept.v1 = {"addresses":["192.168.5.220"],"portRanges":[{"high":8080,"low":8080}],"protocols":["tcp"]}
INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[svc2]
INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:192.168.5.220:8080] service[svc2]

ziti-edge-tunnel is intercepting tcp:192.168.5.56:8080 for svc, which means that it creates a route for this IP through the ziti0 interface. ziti-edge-tunnel is also trying to connect to this address when a ziti client connects, and since the connection is being intercepted you have the classing feedback loop. The same situation exists for svc2 where the IP 192.168.5.220 is being both intercepted and connected to by the same ziti-edge-tunnel. Was this log generated when the policies were different than what's shown in the policy-advisor output here?

10

Thanks for sticking with me and providing the information! It seems that the remote-tunnel is intercepting and hosting the same IP for the svc2 service:

INFO tunnel-cbs:ziti_tunnel_cbs.c:409 new_ziti_intercept() creating intercept for service[svc2] with intercept.v1 = {"addresses":["192.168.5.220"],"portRanges":[{"high":80,"low":80}],"protocols":["tcp"]}
INFO tunnel-cbs:ziti_tunnel_ctrl.c:727 on_service() starting intercepting for service[svc2]
INFO tunnel-cbs:ziti_tunnel_ctrl.c:742 on_service() hosting server_address[tcp:192.168.5.220:8080] service[svc2]

This strongly suggests that the identity running on remote-tunnel (proxmox56) has both dial and bind permission. Here we can see that ziti-edge-tunnel is loading the proxmox56 identity:

INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[proxmox56] connected to controller

And later we see that it received a connection from itself:

INFO tunnel-cbs:ziti_hosting.c:591 on_hosted_client_connect() hosted_service[svc2] client[proxmox56] client_src_addr[tcp:100.64.0.1:37722] dst_addr[tcp:192.168.5.220:8080]: incoming connection

Can you please confirm the policy-advisor still shows the same results as you posted here? Could I also see the role attributes that are assigned to your identities with ziti edge list identities?

Thanks!

11

Fist of all , thanks for your time! i deleted the service "svc" in order to have only the service "svc2"

policy-advisor services
ziti edge policy-advisor services

Policy General Guidelines
  In order for an identity to dial or bind a service, the following must be true:
    - The identity must have access to the service via a service policy of the correct type (dial or bind)
    - The identity must have acces to at least one on-line edge router via an edge router policy
    - The service must have access to at least one on-line edge router via a service edge router policy
    - There must be at least one on-line edge router that both the identity and service have access to.

Policy Advisor Output Guide:
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR.
  ID = identity name
  ID ROUTERS = number of routers accessible to the identity via edge router policies.
    - See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
  SVC = service name
  SVC ROUTERS = number of routers accessible to the service via service edge router policies.
    - See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
  ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
  COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
  DIAL_OK = indicates if the identity has permission to dial the service.
    - See service polices for a service  : ziti edge controller list service service-policies <service>
    - See service polices for an identity: ziti edge controller list identity service-policies <identity>
  BIND_OK = indicates if the identity has permission to bind the service.
  ERROR_LIST = if the status is ERROR, error details will be listed on the following lines

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
OKAY : 10.10.6.13 (1) -> svc2 (1) Common Routers: (1/1) Dial: Y Bind: N

OKAY : proxmox56 (1) -> svc2 (1) Common Routers: (1/1) Dial: N Bind: Y
identities
ziti edge list identities
╭────────────┬──────────────────────────────────────────────────────────────────────┬─────────┬────────────┬─────────────╮
│ ID         │ NAME                                                                 │ TYPE    │ ATTRIBUTES │ AUTH-POLICY │
├────────────┼──────────────────────────────────────────────────────────────────────┼─────────┼────────────┼─────────────┤
│ 60XbOEnHr  │ 10.10.6.13                                                           │ Default │            │ Default     │
│ 8hMVTEY9MG │ ziti-instance-1.europe-west6-a.c.ziticontroller.internal-edge-router │ Router  │            │ Default     │
│ JqiwU4S-s  │ pantelis-host                                                        │ Default │            │ Default     │
│ LyeEO2XHa0 │ proxmox56                                                            │ Default │            │ Default     │
│ zxUgKHypO  │ Default Admin                                                        │ Default │            │ Default     │
╰────────────┴──────────────────────────────────────────────────────────────────────┴─────────┴────────────┴─────────────╯
service policies
ziti edge list service-policies
╭────────────────────────┬───────────┬──────────┬───────────────┬────────────────┬─────────────────────╮
│ ID                     │ NAME      │ SEMANTIC │ SERVICE ROLES │ IDENTITY ROLES │ POSTURE CHECK ROLES │
├────────────────────────┼───────────┼──────────┼───────────────┼────────────────┼─────────────────────┤
│ 2VTjur6skB389r6Sm44gFx │ svc2-Bind │ AnyOf    │ @svc2         │ @proxmox56     │                     │
│ 4LLPrZcHssOuw5itmSDOFj │ svc2-Dial │ AnyOf    │ @svc2         │ @10.10.6.13    │                     │
╰────────────────────────┴───────────┴──────────┴───────────────┴────────────────┴─────────────────────╯

Also the routing table of the remote-tunnel

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.5.254   0.0.0.0         UG    100    0        0 eth0
34.65.229.131   192.168.5.254   255.255.255.255 UGH   100    0        0 eth0
100.64.0.0      0.0.0.0         255.192.0.0     U     0      0        0 ziti0
100.64.0.2      0.0.0.0         255.255.255.255 UH    0      0        0 ziti0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.5.0     0.0.0.0         255.255.255.0   U     100    0        0 eth0
192.168.5.2     0.0.0.0         255.255.255.255 UH    100    0        0 eth0
192.168.5.220   0.0.0.0         255.255.255.255 UH    0      0        0 ziti0
192.168.5.254   0.0.0.0         255.255.255.255 UH    100    0        0 eth0

Is a problem that my identities have not attributes (but as shown in the service-policies the identity-roles are set)?

12

The way you're associating the policies to the identities (by identity) is normally not a problem, but I also noticed that you seem to be using newly enrolled identities presumably after the service policies have been created:

INFO: enrolling /ziti-edge-tunnel/ziti_id.jwt
(7)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
(7)[        0.000]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.36.5 @4271e3e(HEAD) starting at (2024-02-27T16:11:55.409)
(7)[        0.000]    INFO ziti-sdk:ziti_enroll.c:90 ziti_enroll() Ziti C SDK version 0.36.5 @4271e3e(HEAD) starting enrollment at (2024-02-27T16:11:55.409)

Service policies that use identity roles are associated by ID, so if the identities are being recreated then the IDs are going change. In this case (assuming you desire to recreate identities) I think using role attributes will be more predicable, as they approximate "groups" and can be assigned when an identity is created.

If it's not too much trouble I'd like to try an experiment using role attributes instead of identity names. So you'd create the service policies like this:

ziti edge create sp svc2-Bind-by-role-attr Bind --identity-roles '#svc2-binders' --service-roles '@svc2'

and

ziti edge create sp svc2-Dial-by-role-attr Dial --identity-roles '#svc2-dialers' --service-roles '@svc2'

And we'll need to remove the existing policies to prevent them from interfering:

ziti edge delete sp svc2-Bind

and

ziti edge delete sp svc2-Dial

Then you can create (or update) identities with the desired role attributes, and the policies will always be correct even as identities come and go:

ziti edge create identity proxmox56 -a svc2-binders

and

ziti edge create identity 10.10.6.13 -a svc2-dialers
13

Hi,

following your recommendations to use role attributes, we finally achieved a successfull example. However, in another scenario where we continued to use identity names as before, we encountered a similar success. Upon investigation, we discovered a significant difference from our previous experiments: initially, our identities were created with the admin option enabled, but now it's disabled.

This seemingly minor alteration led to unexpected behavior. When we completely removed identities and service-policies but retained the services and services-configs, an interesting observation emerged. Upon recreating identities with the admin configuration enabled—and without configuring any service-policies— ALL the Intercept IPs that included on ther service intercept configs were automatically added to the routing tables on each admin identity.

Nevertheless, thanks a lot for your valuable help!!

1 Like
14

Aha! Now it makes complete sense!

On another note, I started to form a hunch that you might want to connect your endpoints in both directions - e.g. from 10.10.6.13 --> proxmox56 AND from proxmox56 --> 10.10.6.13 - mainly based on your initial setup with two services. If that is your goal, there's a clean way to do that without creating a service for each endpoint. Instead you can use a feature that we call addressable terminators, which we have discussed here.

The example in that thread intercepts hostnames in a wildcard domain, and the hostnames map to the identity names. You could just as easily intercept a subnet and name your identities as IP addresses within that subnet. Feel free to start a new thread if you have questions about this so-called addressable terminator technique.

15

Hello @scareything,

We searched but couldn't find any documentation on the admin option. Can you elaborate on what it does? Why enabling it in an identity adds the intercept IPs in the routing tables?

From your comment (that it makes sense) it seems that this intended behavior? Why is that? Which is the case we may need to use an admin identity?

Thanks a lot!

16

Hi,

Yes I should have elaborated. Thanks for asking! In the early days of openziti before the tunnelers and their feature sets were fully developed, it was decided that admin identities would have full permissions to everything, including bind and dual permissions to services. Your situation has actually inspired us to rethink that decision.

You generally only want admin identities for managing your network.