I'm currently working on a proof-of-concept for OpenZiti on a controlled network (e.g. some hosts eventually won't have internet access, so using the built-in PKI), and I'm having a bit of trouble getting a tunnel to connect to a router. All the components of this OZ network are run inside docker containers, but on one or more physically separate machines.
The basic design is something like this, where VMs 2/3 have all non-OZ ports blocked, and no internet access beyond via VM1.
VM 1: Controller
VM 2: Router + Tunnel + Other software
VM 3: Router + Tunnel + Other software
I've got a controller and two routers setup, that all seem to be communicating properly, but now need to add in a tunnel to one router. I've been following the steps at Containers | OpenZiti and am currently using the following docker-compose for the tunnel (the same docker compose has logic for the router, but not included here);
ziti-tunnel:
image: openziti/ziti-host:0.22.12
env_file:
- ./.env
environment:
- ZITI_IDENTITY_BASENAME=${ZITI_TUNNEL_NAME}
- ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}
network_mode: host
user: "2171:2171"
extra_hosts:
- "ziti-edge-controller:X.X.X.163"
- "ziti-edge-router-1:X.X.X.161"
volumes:
- ziti-fs:/ziti-edge-tunnel
The ziti-fs is the same volume mounted to "ziti-edge-router-1", which is running on the same machine. I've manually created the identity JWT in that volume from inside the edge router, then started the tunnel. It's successfully talked to the controller and enrolled, as a valid .json was written to the volume by the tunnel too.
However, that's the end of the successes - from here the tunnel just starts continuously retrying to reach the edge router;
oz-ziti-tunnel-1 | WARN: the identities directory is only available inside this container because /ziti-edge-tunnel is not a mounted volume. Be careful to not publish this image with identity inside or lose access to the identity by removing the image prematurely.
oz-ziti-tunnel-1 | INFO: setting NF_REG_NAME to ${ZITI_IDENTITY_BASENAME} (ziti-tunnel-1)
oz-ziti-tunnel-1 | DEBUG: waiting 1s for /ziti-edge-tunnel/ziti-tunnel-1.json (or token) to appear
oz-ziti-tunnel-1 | INFO: found identity file /ziti-edge-tunnel/ziti-tunnel-1.json
oz-ziti-tunnel-1 | DEBUG: evaluating positionals: run-host
oz-ziti-tunnel-1 | INFO: running ziti-edge-tunnel
oz-ziti-tunnel-1 | (8)[ 0.000] INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=3/INFO
oz-ziti-tunnel-1 | (8)[ 0.000] INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.35.4 @9756522(HEAD) starting at (2023-10-24T13:43:17.215)
oz-ziti-tunnel-1 | (8)[ 0.000] INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.22.12)
oz-ziti-tunnel-1 | (8)[ 0.000] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1608 make_socket_path() local 'ziti' group not found.
oz-ziti-tunnel-1 | (8)[ 0.000] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1609 make_socket_path() please create the 'ziti' group by running these commands:
oz-ziti-tunnel-1 | (8)[ 0.000] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1611 make_socket_path() sudo groupadd --system ziti
oz-ziti-tunnel-1 | (8)[ 0.000] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1612 make_socket_path() users can then be added to the 'ziti' group with:
oz-ziti-tunnel-1 | (8)[ 0.000] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1613 make_socket_path() sudo usermod --append --groups ziti <USER>
oz-ziti-tunnel-1 | (8)[ 0.000] WARN ziti-edge-tunnel:ziti-edge-tunnel.c:1712 run_tunneler_loop() One or more socket servers did not properly start.
oz-ziti-tunnel-1 | (8)[ 0.001] INFO tunnel-cbs:ziti_tunnel_ctrl.c:907 load_ziti_async() attempting to load ziti instance[/ziti-edge-tunnel/ziti-tunnel-1.json]
oz-ziti-tunnel-1 | (8)[ 0.001] INFO tunnel-cbs:ziti_tunnel_ctrl.c:914 load_ziti_async() loading ziti instance[/ziti-edge-tunnel/ziti-tunnel-1.json]
oz-ziti-tunnel-1 | (8)[ 0.001] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1117 load_id_cb() identity[/ziti-edge-tunnel/ziti-tunnel-1.json] loaded
oz-ziti-tunnel-1 | (8)[ 0.001] WARN ziti-edge-tunnel:instance.c:40 find_tunnel_identity() Identity ztx[/ziti-edge-tunnel/ziti-tunnel-1.json] is not loaded yet or already removed.
oz-ziti-tunnel-1 | (8)[ 0.001] ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
oz-ziti-tunnel-1 | (8)[ 0.001] ERROR ziti-edge-tunnel:instance-config.c:142 save_tunnel_status_to_file() Could not open config file /var/lib/ziti/config.json to store the tunnel status data
oz-ziti-tunnel-1 | (8)[ 0.003] INFO ziti-sdk:ziti.c:449 ziti_init_async() ztx[0] using tlsuv[v0.26.1], tls[mbed TLS 2.28.1]
oz-ziti-tunnel-1 | (8)[ 0.003] INFO ziti-sdk:ziti.c:450 ziti_init_async() ztx[0] Loading ziti context with controller[https://ziti-edge-controller:1280]
oz-ziti-tunnel-1 | (8)[ 0.003] INFO ziti-sdk:ziti.c:913 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://ziti-edge-controller:1280] api_session_status[0] api_session_expired[TRUE]
oz-ziti-tunnel-1 | (8)[ 0.034] INFO ziti-sdk:ziti.c:1671 version_cb() ztx[0] connected to controller https://ziti-edge-controller:1280 version v0.30.5(4f324bd22875 2023-10-13T20:22:56Z)
oz-ziti-tunnel-1 | (8)[ 0.038] INFO ziti-sdk:ziti.c:1560 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
oz-ziti-tunnel-1 | (8)[ 0.038] INFO tunnel-cbs:ziti_tunnel_ctrl.c:767 on_ziti_event() ziti_ctx[ziti-tunnel-1] connected to controller
oz-ziti-tunnel-1 | (8)[ 0.038] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1147 on_event() ztx[/ziti-edge-tunnel/ziti-tunnel-1.json] context event : status is OK
oz-ziti-tunnel-1 | (8)[ 0.041] INFO ziti-sdk:channel.c:237 new_ziti_channel() ch[0] (ziti-edge-router-1) new channel for ztx[0] identity[ziti-tunnel-1]
oz-ziti-tunnel-1 | (8)[ 0.041] INFO tunnel-cbs:ziti_tunnel_ctrl.c:839 on_ziti_event() ztx[ziti-tunnel-1] added edge router ziti-edge-router-1@ziti-edge-router
oz-ziti-tunnel-1 | (8)[ 0.041] INFO ziti-sdk:channel.c:736 reconnect_channel() ch[0] reconnecting NOW
oz-ziti-tunnel-1 | (8)[ 0.041] INFO ziti-sdk:channel.c:736 reconnect_channel() ch[0] reconnecting NOW
oz-ziti-tunnel-1 | (8)[ 0.046] ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-edge-router-1] [-3008/unknown node or service]
oz-ziti-tunnel-1 | (8)[ 0.046] INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting in 5214ms (attempt = 1)
oz-ziti-tunnel-1 | (8)[ 1.040] INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
oz-ziti-tunnel-1 | (8)[ 5.277] ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-edge-router-1] [-3008/unknown node or service]
oz-ziti-tunnel-1 | (8)[ 5.277] INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting in 1641ms (attempt = 2)
oz-ziti-tunnel-1 | (8)[ 6.934] ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[0] failed to connect to ER[ziti-edge-router-1] [-3008/unknown node or service]
oz-ziti-tunnel-1 | (8)[ 6.934] INFO ziti-sdk:channel.c:733 reconnect_channel() ch[0] reconnecting in 25689ms (attempt = 3)
I've confirmed that I can reach the edge router using the hostname "ziti-edge-router-1" on that machine, same way I can reach the controller. Both curl ziti-controller:1280 and curl ziti-edge-router-1:3022 return an empty response, but they do return rather than hanging/timing out.
I know there are some warnings about being unable to backup configs/write logs, I'm not sure whether they're relevant. Other posts I could find with similar errors ended up being down to a failure in DNS resolution, however (as curl shows) the hostname seems correct here, and is definitely present in /etc/hosts inside the container.
Does anyone have any suggestions of other angles to try, or anything I can do to shed more light on exactly what "unknown node or service" means it's unable to do?
