Published Tunnel Image Not Working As Expected

1

Hello All,

I'm trying to get docker based tunnelers working. I'll be using Docker in production so I need to dogfood things locally.

The published image appears to have an issue that does not occur if I handroll an image.

Common Steps

Create Identity

Nothing special here.

Run the quick Launch

cd ../openziti
docker compose up

Your reds/blue and purple should be up before proceeding.

Grab the IPS from the purple containers

docker container inspect open_ziti-ziti-edge-router-1
docker container inspect open_ziti-ziti-edge-controller-1

What doesn't work with published image

Follow guide here - Containers | OpenZiti

Create your compose

Plugin IP to the extra_hosts field

version: '2.4'
services:
ziti-tun:
image: openziti/ziti-edge-tunnel
devices:
- /dev/net/tun:/dev/net/tun
volumes:
- ziti-tunnel-fs:/ziti-edge-tunnel
- /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
environment:
- ZITI_IDENTITY_BASENAME=${OFFLINE_MAPPING_BASENAME}
- ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}
- ZITI_IDENTITY_WAIT=180
extra_hosts:
- "ziti-edge-controller:172.24.0.2"
- "ziti-edge-router:172.24.0.6"
network_mode: host
privileged: true
volumes:
ziti-tunnel-fs:

Drop into the container

docker compose run -it --entrypoint bash ziti-tun

Try enroll

This essentially does what the docker entrypoint script does

echo $ZITI_ENROLL_TOKEN > /tmp/indocker.client.jwt
[root@LOCLAP680 /]# ziti-edge-tunnel enroll --jwt /tmp/indocker.client.jwt --identity /tmp/indocker.client.json
(11)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(11)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(11)[ 0.000] INFO ziti-sdk:ziti_enroll.c:92 ziti_enroll() Ziti C SDK version 0.33.2 @e06b76c(HEAD) starting enrollment at (2023-08-16T23:07:55.326)
(11)[ 0.027] ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[ziti-edge-controller] request failed: -103(software caused connection abort)
(11)[ 0.027] ERROR ziti-sdk:ziti_enroll.c:235 enroll_cb() failed to enroll with controller: https://ziti-edge-controller:1280 CONTROLLER_UNAVAILABLE (software caused connection abort)
(11)[ 0.027] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2137 enroll_cb() enrollment failed: CONTROLLER_UNAVAILABLE(-3)

Controller is unhappy with TLS

open_ziti-ziti-controller-1 | [2994.808] INFO : http: TLS handshake error from 172.24.0.1:45040: local error: tls: bad record MAC

DEBUG STEPS/What does work.

Essentially all we are doing is creating a new image and handrolling the tunneler. The docker compose file doesn't change except the image name.

The markdown editor is esacaping the backslashes. So if you try to run this the Lines in the RUN command need a backslash

Create Debug Image

from ubuntu:22.04
RUN apt update &&
apt install -y netcat nano gpg curl &&
curl -sSLf https://get.openziti.io/tun/package-repos.gpg | gpg --dearmor --output /usr/share/keyrings/openziti.gpg &&
echo "deb [signed-by=/usr/share/keyrings/openziti.gpg] JFrog jammy main" | tee /etc/apt/sources.list.d/openziti.list >/dev/null &&
apt update &&
apt install -y ziti-edge-tunnel

Create your Docker Compose Service

Change image name, plug the IPS you looked up into the extra_hosts

version: '2.4'
services:
ziti-tun:
image: debug-ziti-tunneler:latest
devices:
- /dev/net/tun:/dev/net/tun
volumes:
- ziti-tunnel-fs:/ziti-edge-tunnel
- /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
environment:
- ZITI_IDENTITY_BASENAME=${OFFLINE_MAPPING_BASENAME}
- ZITI_ENROLL_TOKEN=${ZITI_ENROLL_TOKEN}
- ZITI_IDENTITY_WAIT=180
extra_hosts:
- "ziti-edge-controller:172.24.0.2"
- "ziti-edge-router:172.24.0.6"
network_mode: host
privileged: true
volumes:
ziti-tunnel-fs:

Run the Docker Compose Service

This just gets us a running container so we can do things by hand

docker compose run -it --entrypoint bash ziti-tun

Steps to Run on the Container

root@LOCLAP680:/# echo $ZITI_ENROLL_TOKEN >> /tmp/indocker.client.jwt
root@LOCLAP680:/# ziti-edge-tunnel enroll --jwt /tmp/indocker.client.jwt --identity /tmp/indocker.client.json
(9)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(9)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(9)[ 0.000] INFO ziti-sdk:ziti_enroll.c:90 ziti_enroll() Ziti C SDK version 0.32.8 @d7f329f(HEAD) starting enrollment at (2023-08-16T22:58:07.479)
root@LOCLAP680:/# ziti-edge-tunnel run --identity /tmp/indocker.client.json
(14)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(14)[ 0.000] INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.21.5-local)
(14)[ 0.000] INFO tunnel-cbs:ziti_dns.c:168 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(14)[ 0.000] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1621 make_socket_path() effective group set to 'ziti' (gid=999)
(14)[ 0.009] INFO tunnel-cbs:ziti_tunnel_ctrl.c:865 load_ziti_async() attempting to load ziti instance from file[/tmp/indocker.client.json]
(14)[ 0.009] INFO tunnel-cbs:ziti_tunnel_ctrl.c:872 load_ziti_async() loading ziti instance from /tmp/indocker.client.json
(14)[ 0.009] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(14)[ 0.009] INFO ziti-edge-tunnel:resolvers.c:67 init_libsystemd() Initializing libsystemd
(14)[ 0.009] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1114 load_id_cb() identity[/tmp/indocker.client.json] loaded
(14)[ 0.009] WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[/tmp/indocker.client.json] is not loaded yet or already removed.
(14)[ 0.009] ERROR ziti-edge-tunnel:instance-config.c:136 save_tunnel_status_to_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment
(14)[ 0.010] INFO ziti-sdk:ziti.c:407 ziti_init_async() ztx[0] Ziti C SDK version 0.32.8 @d7f329f(HEAD) starting at (2023-08-16T22:58:14.688)
(14)[ 0.010] INFO ziti-sdk:ziti.c:410 ziti_init_async() ztx[0] using tlsuv[v0.23.8], tls[mbed TLS 2.28.1]
(14)[ 0.010] INFO ziti-sdk:ziti.c:412 ziti_init_async() ztx[0] Loading ziti context with controller[https://ziti-edge-controller:1280]
(14)[ 0.010] INFO ziti-sdk:ziti.c:848 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctlr[https://ziti-edge-controller:1280] api_session_status[0] api_session_expired[TRUE]
(14)[ 0.010] INFO ziti-edge-tunnel:resolvers.c:355 try_libsystemd_resolver() systemd-resolved selected as dns resolver manager
(14)[ 0.013] INFO ziti-edge-tunnel:resolvers.c:399 dns_update_systemd_resolved() Setting wildcard routing only domain on interface: tun0
(14)[ 0.033] INFO ziti-sdk:ziti.c:1507 version_cb() ztx[0] connected to controller https://ziti-edge-controller:1280 version v0.29.0(3ca2dd2f4e7b 2023-07-13T15:53:37Z)
(14)[ 0.048] INFO ziti-sdk:ziti.c:1398 ziti_set_api_session() ztx[0] api session set, setting api_session_timer to 1740s
(14)[ 0.048] INFO tunnel-cbs:ziti_tunnel_ctrl.c:726 on_ziti_event() ziti_ctx[indocker] connected to controller
(14)[ 0.048] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1144 on_event() ztx[/tmp/indocker.client.json] context event : status is OK
(14)[ 0.051] INFO ziti-sdk:channel.c:234 new_ziti_channel() ch[0] (ziti-edge-router@tls://ziti-edge-router:3022) new channel for ztx[0] identity[indocker]
(14)[ 0.051] INFO tunnel-cbs:ziti_tunnel_ctrl.c:798 on_ziti_event() ztx[indocker] added edge router ziti-edge-router@tls://ziti-edge-router:3022@ziti-edge-router
(14)[ 0.051] INFO ziti-sdk:channel.c:728 reconnect_channel() ch[0] reconnecting NOW
(14)[ 0.086] INFO ziti-sdk:channel.c:626 hello_reply_cb() ch[0] connected. EdgeRouter version: v0.29.0|3ca2dd2f4e7b|2023-07-13T15:53:37Z|linux|amd64
(14)[ 0.086] INFO tunnel-cbs:ziti_tunnel_ctrl.c:802 on_ziti_event() ztx[indocker] router ziti-edge-router@tls://ziti-edge-router:3022 connected
(14)[ 1.048] INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected

The logs look good and I can see the identity connected in ZAC/UI

1 Like
2

Hey @bearrito!

I see that you were able to to test the tunneler in run mode as an intercepting proxy with ziti-edge-tunnel installed in your custom container image, but for some reason it didn’t work with the official image.

I have definitely used the Ziti tunneler running in this container image, both in run mode and run-host mode, for an intercepting proxy and hosting-only reverse proxy, respectively.

I think I’m seeing the same thing you reported. The tunneler connects to the controller, gets a list of routers, then a certificate verification fails when it connects to the first router.

handshake error error:0A000086:SSL routines::certificate verify failed

Here’s how I enabled the debug output:

docker compose run --env TLSUV_DEBUG=4 ziti-tun run --verbose=4

I’ll keep looking at precisely why that handshake failed. I didn’t encounter the controller TLS handshake failure you reported at all, so far it’s just the router in my test env.

I suspect the reason it worked in your custom container image is that you installed the binary from the DEB package, and that is built separately for each target OS.

We released the 0.22.5 DEB package today, and I’m curious to learn if you have the same success with that version in your custom container image. I triggered the same handshake failure with that version by installing the Ubuntu Jammy DEB.

To narrow down the variables it will be helpful to pin some container and binary versions so we know exactly what we’re dealing with.

To be cont’d…

3

Here’s the Dockerfile I used to pin the version that worked for both of us.

FROM ubuntu:22.04
RUN apt-get update \
    && apt-get install -y netcat nano gpg curl \
    && curl -sSLf https://get.openziti.io/tun/package-repos.gpg \
        | gpg --dearmor --output /usr/share/keyrings/openziti.gpg \
    && chmod +r /usr/share/keyrings/openziti.gpg \
    && echo 'deb [signed-by=/usr/share/keyrings/openziti.gpg] https://packages.openziti.org/zitipax-openziti-deb-stable jammy main' \
        > /etc/apt/sources.list.d/openziti.list \
    && apt-get update \
    && apt-get install -y ziti-edge-tunnel=0.21.5 \
    && apt-mark hold ziti-edge-tunnel \
    && apt-get --yes autoremove \
    && apt-get clean autoclean \
    && rm -fr /var/lib/apt/lists/{apt,dpkg,cache,log} /tmp/* /var/tmp/*
4

@bearrito We found a problem with the quickstart version that was current at the time you wrote this topic, so I think the best way to proceed is to start fresh with the latest Compose quickstart version by setting in your .env file in the Compose project directory ZITI_VERSION=0.30.0. This var determines the openziti/quickstart container image tag in docker-compose.yml.