Tunnel unable to reach local router

I'm currently working on a proof-of-concept for OpenZiti on a controlled network (e.g. some hosts eventually won't have internet access, so using the built-in PKI), and I'm having a bit of trouble getting a tunnel to connect to a router. All the components of this OZ network are run inside docker containers, but on one or more physically separate machines.
The basic design is something like this, where VMs 2/3 have all non-OZ ports blocked, and no internet access beyond via VM1.
VM 1: Controller
VM 2: Router + Tunnel + Other software
VM 3: Router + Tunnel + Other software

I've got a controller and two routers setup, that all seem to be communicating properly, but now need to add in a tunnel to one router. I've been following the steps at Containers | OpenZiti and am currently using the following docker-compose for the tunnel (the same docker compose has logic for the router, but not included here);

    image: openziti/ziti-host:0.22.12
      - ./.env
    network_mode: host
    user: "2171:2171"
      - "ziti-edge-controller:X.X.X.163"
      - "ziti-edge-router-1:X.X.X.161"
      - ziti-fs:/ziti-edge-tunnel

The ziti-fs is the same volume mounted to "ziti-edge-router-1", which is running on the same machine. I've manually created the identity JWT in that volume from inside the edge router, then started the tunnel. It's successfully talked to the controller and enrolled, as a valid .json was written to the volume by the tunnel too.
However, that's the end of the successes - from here the tunnel just starts continuously retrying to reach the edge router;

oz-ziti-tunnel-1  | WARN: the identities directory is only available inside this container because /ziti-edge-tunnel is not a mounted volume. Be careful to not publish this image with identity inside or lose access to the identity by removing the image prematurely.
oz-ziti-tunnel-1  | INFO: setting NF_REG_NAME to ${ZITI_IDENTITY_BASENAME} (ziti-tunnel-1)
oz-ziti-tunnel-1  | DEBUG: waiting 1s for /ziti-edge-tunnel/ziti-tunnel-1.json (or token) to appear
oz-ziti-tunnel-1  | INFO: found identity file /ziti-edge-tunnel/ziti-tunnel-1.json
oz-ziti-tunnel-1  | DEBUG: evaluating positionals: run-host
oz-ziti-tunnel-1  | INFO: running ziti-edge-tunnel
oz-ziti-tunnel-1  | (8)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
oz-ziti-tunnel-1  | (8)[        0.000]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.35.4 @9756522(HEAD) starting at (2023-10-24T13:43:17.215)
oz-ziti-tunnel-1  | (8)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.12)
oz-ziti-tunnel-1  | (8)[        0.000]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1608 make_socket_path() local 'ziti' group not found.
oz-ziti-tunnel-1  | (8)[        0.000]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1609 make_socket_path() please create the 'ziti' group by running these commands:
oz-ziti-tunnel-1  | (8)[        0.000]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1611 make_socket_path() sudo groupadd --system ziti
oz-ziti-tunnel-1  | (8)[        0.000]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1612 make_socket_path() users can then be added to the 'ziti' group with:
oz-ziti-tunnel-1  | (8)[        0.000]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1613 make_socket_path() sudo usermod --append --groups ziti <USER>
oz-ziti-tunnel-1  | (8)[        0.000]    WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1712 run_tunneler_loop() One or more socket servers did not properly start.
oz-ziti-tunnel-1  | (8)[        0.001]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/ziti-edge-tunnel/ziti-tunnel-1.json]
oz-ziti-tunnel-1  | (8)[        0.001]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/ziti-edge-tunnel/ziti-tunnel-1.json]
oz-ziti-tunnel-1  | (8)[        0.001]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1117 load_id_cb() identity[/ziti-edge-tunnel/ziti-tunnel-1.json] loaded
oz-ziti-tunnel-1  | (8)[        0.001]    WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/ziti-edge-tunnel/ziti-tunnel-1.json] is not loaded yet or already removed.
oz-ziti-tunnel-1  | (8)[        0.001]   ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
oz-ziti-tunnel-1  | (8)[        0.001]   ERROR ziti-edge-tunnel:instance-config.c:142 save_tunnel_status_to_file() Could not open config file /var/lib/ziti/config.json to store the tunnel status data
oz-ziti-tunnel-1  | (8)[        0.003]    INFO ziti-sdk:ziti.c:449 ziti_init_async() ztx[0] using tlsuv[v0.26.1], tls[mbed TLS 2.28.1]
oz-ziti-tunnel-1  | (8)[        0.003]    INFO ziti-sdk:ziti.c:450 ziti_init_async() ztx[0] Loading ziti context with controller[https://ziti-edge-controller:1280]
oz-ziti-tunnel-1  | (8)[        0.003]    INFO ziti-sdk:ziti.c:913 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://ziti-edge-controller:1280] api_session_status[0] api_session_expired[TRUE]
oz-ziti-tunnel-1  | (8)[        0.034]    INFO ziti-sdk:ziti.c:1671 version_cb() ztx[0] connected to controller https://ziti-edge-controller:1280 version v0.30.5(4f324bd22875 2023-10-13T20:22:56Z)
oz-ziti-tunnel-1  | (8)[        0.038]    INFO ziti-sdk:ziti.c:1560 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
oz-ziti-tunnel-1  | (8)[        0.038]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[ziti-tunnel-1] connected to controller
oz-ziti-tunnel-1  | (8)[        0.038]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1147 on_event() ztx[/ziti-edge-tunnel/ziti-tunnel-1.json] context event : status is OK
oz-ziti-tunnel-1  | (8)[        0.041]    INFO ziti-sdk:channel.c:237 new_ziti_channel() ch[0] (ziti-edge-router-1) new channel for ztx[0] identity[ziti-tunnel-1]
oz-ziti-tunnel-1  | (8)[        0.041]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[ziti-tunnel-1] added edge router ziti-edge-router-1@ziti-edge-router
oz-ziti-tunnel-1  | (8)[        0.041]    INFO ziti-sdk:channel.c:736 reconnect_channel() ch[0] reconnecting NOW
oz-ziti-tunnel-1  | (8)[        0.041]    INFO ziti-sdk:channel.c:736 reconnect_channel() ch[0] reconnecting NOW
oz-ziti-tunnel-1  | (8)[        0.046]   ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-edge-router-1] [-3008/unknown node or service]
oz-ziti-tunnel-1  | (8)[        0.046]    INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting in 5214ms (attempt = 1)
oz-ziti-tunnel-1  | (8)[        1.040]    INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
oz-ziti-tunnel-1  | (8)[        5.277]   ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-edge-router-1] [-3008/unknown node or service]
oz-ziti-tunnel-1  | (8)[        5.277]    INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting in 1641ms (attempt = 2)
oz-ziti-tunnel-1  | (8)[        6.934]   ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-edge-router-1] [-3008/unknown node or service]
oz-ziti-tunnel-1  | (8)[        6.934]    INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting in 25689ms (attempt = 3)

I've confirmed that I can reach the edge router using the hostname "ziti-edge-router-1" on that machine, same way I can reach the controller. Both curl ziti-controller:1280 and curl ziti-edge-router-1:3022 return an empty response, but they do return rather than hanging/timing out.

I know there are some warnings about being unable to backup configs/write logs, I'm not sure whether they're relevant. Other posts I could find with similar errors ended up being down to a failure in DNS resolution, however (as curl shows) the hostname seems correct here, and is definitely present in /etc/hosts inside the container.

Does anyone have any suggestions of other angles to try, or anything I can do to shed more light on exactly what "unknown node or service" means it's unable to do?

Hi @Coconutcoo, welcome to the community and to OpenZiti!

My guess is that your routers have advertised addresses that are incorrect. Can you open the router config files and look at the listeners.binding=edge.options.advertise address? For example one of mine looks like this:

# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:
      advertise: my.dns.entry:8442

The value in advertise - can you connect to it from the tunneler? I'm guessing it's the docker 'instance' hostname (example: ab251232) in there, not the docker container name (ziti-edge-router-1)

1 Like

Aha, that's likely it - I had the following;

# bindings of edge and tunnel requires an "edge" section below
  - binding: edge
    address: tls:
      advertise: ziti-edge-router:3022
      connectTimeoutMs: 5000
      getSessionTimeout: 60
  - binding: tunnel
      mode: host #tproxy|host

I couldn't connect to ziti-edge-router:3022, since it wasn't in the extra_hosts block. (and that wasn't what the tunnel was looking for).

I think I'd previously changed the edge router name in the .env without editing/regenerating the config file for the router. Once I did amended that to ziti-edge-router-1:8442 everything seems to work fine.

Thanks for your help there! As a bit of a follow up, is there any issue beyond organisation with having multiple OZ components with the same name? I'm looking to essentially templatise part of the ziti network (a router/tunnel pair) and possibly copy that over to more VMs, if 20 routers all have the name "ziti-edge-router" and 20 tunnels called "ziti-tunnel", will that cause issues?

Names are unique within an object type group (router, identity, etc). If you're crossing over types, personally I just get confused easily so I tend to make the name unique globally for my puny human :brain: :laughing:

1 Like