# Published Tunnel Image Not Working As Expected

**URL:** <https://openziti.discourse.group/t/published-tunnel-image-not-working-as-expected/1516>\
**Category:** Uncategorized\
**Created:** [August 16, 2023, 11:17pm UTC](https://openziti.discourse.group/t/published-tunnel-image-not-working-as-expected/1516 "2023-08-16T23:17:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![bearrito](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/bearrito/32/1189_2.png) [@bearrito](https://openziti.discourse.group/u/bearrito)\
**Post date:** [August 16, 2023, 11:17pm UTC](https://openziti.discourse.group/t/published-tunnel-image-not-working-as-expected/1516/1 "2023-08-16T23:17:16Z")

</div>

Hello All,

I'm trying to get docker based tunnelers working. I'll be using Docker in production so I need to dogfood things locally.

The published image appears to have an issue that does not occur if I handroll an image.

# Common Steps

# Create Identity

Nothing special here.

# Run the quick Launch

> cd ../openziti  
> docker compose up

Your reds/blue and purple should be up before proceeding.

# Grab the IPS from the purple containers

> docker container inspect open\_ziti-ziti-edge-router-1  
> docker container inspect open\_ziti-ziti-edge-controller-1

# What doesn't work with published image

Follow guide here - [NetFoundry Docs Portal](https://openziti.io/docs/reference/tunnelers/linux/container/)

# Create your compose

Plugin IP to the extra\_hosts field

> version: '2.4'  
> services:  
> ziti-tun:  
> image: openziti/ziti-edge-tunnel  
> devices:  
> - /dev/net/tun:/dev/net/tun  
> volumes:  
> - ziti-tunnel-fs:/ziti-edge-tunnel  
> - /var/run/dbus/system\_bus\_socket:/var/run/dbus/system\_bus\_socket  
> environment:  
> - ZITI\_IDENTITY\_BASENAME=${OFFLINE\_MAPPING\_BASENAME}  
> - ZITI\_ENROLL\_TOKEN=${ZITI\_ENROLL\_TOKEN}  
> - ZITI\_IDENTITY\_WAIT=180  
> extra\_hosts:  
> - "ziti-edge-controller:172.24.0.2"  
> - "ziti-edge-router:172.24.0.6"  
> network\_mode: host  
> privileged: true  
> volumes:  
> ziti-tunnel-fs:

# Drop into the container

> docker compose run -it --entrypoint bash ziti-tun

# Try enroll

This essentially does what the docker entrypoint script does

> echo $ZITI\_ENROLL\_TOKEN \> /tmp/indocker.client.jwt  
> [root@LOCLAP680 /]# ziti-edge-tunnel enroll --jwt /tmp/indocker.client.jwt --identity /tmp/indocker.client.json  
> (11)[0.000] INFO ziti-sdk:utils.c:188 ziti\_log\_set\_level() set log level: root=3/INFO  
> (11)[0.000] INFO ziti-sdk:utils.c:188 ziti\_log\_set\_level() set log level: root=3/INFO  
> (11)[0.000] INFO ziti-sdk:ziti\_enroll.c:92 ziti\_enroll() Ziti C SDK version 0.33.2 @e06b76c(HEAD) starting enrollment at (2023-08-16T23:07:55.326)  
> (11)[0.027] ERROR ziti-sdk:ziti\_ctrl.c:154 ctrl\_resp\_cb() ctrl[ziti-edge-controller] request failed: -103(software caused connection abort)  
> (11)[0.027] ERROR ziti-sdk:ziti\_enroll.c:235 enroll\_cb() failed to enroll with controller: [https://ziti-edge-controller:1280](https://ziti-edge-controller:1280) CONTROLLER\_UNAVAILABLE (software caused connection abort)  
> (11)[0.027] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2137 enroll\_cb() enrollment failed: CONTROLLER\_UNAVAILABLE(-3)

# Controller is unhappy with TLS

> open\_ziti-ziti-controller-1 | [2994.808] INFO : http: TLS handshake error from 172.24.0.1:45040: local error: tls: bad record MAC

# DEBUG STEPS/What does work.

Essentially all we are doing is creating a new image and handrolling the tunneler. The docker compose file doesn't change except the image name.

The markdown editor is esacaping the backslashes. So if you try to run this the Lines in the RUN command need a backslash

# Create Debug Image

> from ubuntu:22.04  
> RUN apt update &&   
> apt install -y netcat nano gpg curl &&   
> curl -sSLf [https://get.openziti.io/tun/package-repos.gpg](https://get.openziti.io/tun/package-repos.gpg) | gpg --dearmor --output /usr/share/keyrings/openziti.gpg &&   
> echo "deb [signed-by=/usr/share/keyrings/openziti.gpg] [JFrog](https://packages.openziti.org/zitipax-openziti-deb-stable) jammy main" | tee /etc/apt/sources.list.d/openziti.list \>/dev/null &&   
> apt update &&   
> apt install -y ziti-edge-tunnel

# Create your Docker Compose Service

Change image name, plug the IPS you looked up into the extra\_hosts

> version: '2.4'  
> services:  
> ziti-tun:  
> image: debug-ziti-tunneler:latest  
> devices:  
> - /dev/net/tun:/dev/net/tun  
> volumes:  
> - ziti-tunnel-fs:/ziti-edge-tunnel  
> - /var/run/dbus/system\_bus\_socket:/var/run/dbus/system\_bus\_socket  
> environment:  
> - ZITI\_IDENTITY\_BASENAME=${OFFLINE\_MAPPING\_BASENAME}  
> - ZITI\_ENROLL\_TOKEN=${ZITI\_ENROLL\_TOKEN}  
> - ZITI\_IDENTITY\_WAIT=180  
> extra\_hosts:  
> - "ziti-edge-controller:172.24.0.2"  
> - "ziti-edge-router:172.24.0.6"  
> network\_mode: host  
> privileged: true  
> volumes:  
> ziti-tunnel-fs:

# Run the Docker Compose Service

This just gets us a running container so we can do things by hand

> docker compose run -it --entrypoint bash ziti-tun

# Steps to Run on the Container

> root@LOCLAP680:/# echo $ZITI\_ENROLL\_TOKEN \>\> /tmp/indocker.client.jwt  
> root@LOCLAP680:/# ziti-edge-tunnel enroll --jwt /tmp/indocker.client.jwt --identity /tmp/indocker.client.json  
> (9)[0.000] INFO ziti-sdk:utils.c:188 ziti\_log\_set\_level() set log level: root=3/INFO  
> (9)[0.000] INFO ziti-sdk:utils.c:188 ziti\_log\_set\_level() set log level: root=3/INFO  
> (9)[0.000] INFO ziti-sdk:ziti\_enroll.c:90 ziti\_enroll() Ziti C SDK version 0.32.8 @d7f329f(HEAD) starting enrollment at (2023-08-16T22:58:07.479)  
> root@LOCLAP680:/# ziti-edge-tunnel run --identity /tmp/indocker.client.json  
> (14)[0.000] INFO ziti-sdk:utils.c:188 ziti\_log\_set\_level() set log level: root=3/INFO  
> (14)[0.000] INFO tunnel-sdk:ziti\_tunnel.c:60 create\_tunneler\_ctx() Ziti Tunneler SDK (v0.21.5-local)  
> (14)[0.000] INFO tunnel-cbs:ziti\_dns.c:168 seed\_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)  
> (14)[0.000] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1621 make\_socket\_path() effective group set to 'ziti' (gid=999)  
> (14)[0.009] INFO tunnel-cbs:ziti\_tunnel\_ctrl.c:865 load\_ziti\_async() attempting to load ziti instance from file[/tmp/indocker.client.json]  
> (14)[0.009] INFO tunnel-cbs:ziti\_tunnel\_ctrl.c:872 load\_ziti\_async() loading ziti instance from /tmp/indocker.client.json  
> (14)[0.009] INFO ziti-sdk:utils.c:188 ziti\_log\_set\_level() set log level: root=3/INFO  
> (14)[0.009] INFO ziti-edge-tunnel:resolvers.c:67 init\_libsystemd() Initializing libsystemd  
> (14)[0.009] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1114 load\_id\_cb() identity[/tmp/indocker.client.json] loaded  
> (14)[0.009] WARN ziti-edge-tunnel:instance.c:39 find\_tunnel\_identity() Identity ztx[/tmp/indocker.client.json] is not loaded yet or already removed.  
> (14)[0.009] ERROR ziti-edge-tunnel:instance-config.c:136 save\_tunnel\_status\_to\_file() Could not copy config file [/var/lib/ziti/config.json] to backup config file, the config might not exists at the moment  
> (14)[0.010] INFO ziti-sdk:ziti.c:407 ziti\_init\_async() ztx[0] Ziti C SDK version 0.32.8 @d7f329f(HEAD) starting at (2023-08-16T22:58:14.688)  
> (14)[0.010] INFO ziti-sdk:ziti.c:410 ziti\_init\_async() ztx[0] using tlsuv[v0.23.8], tls[mbed TLS 2.28.1]  
> (14)[0.010] INFO ziti-sdk:ziti.c:412 ziti\_init\_async() ztx[0] Loading ziti context with controller[[https://ziti-edge-controller:1280](https://ziti-edge-controller:1280)]  
> (14)[0.010] INFO ziti-sdk:ziti.c:848 ziti\_re\_auth\_with\_cb() ztx[0] starting to re-auth with ctlr[[https://ziti-edge-controller:1280](https://ziti-edge-controller:1280)] api\_session\_status[0] api\_session\_expired[TRUE]  
> (14)[0.010] INFO ziti-edge-tunnel:resolvers.c:355 try\_libsystemd\_resolver() systemd-resolved selected as dns resolver manager  
> (14)[0.013] INFO ziti-edge-tunnel:resolvers.c:399 dns\_update\_systemd\_resolved() Setting wildcard routing only domain on interface: tun0  
> (14)[0.033] INFO ziti-sdk:ziti.c:1507 version\_cb() ztx[0] connected to controller [https://ziti-edge-controller:1280](https://ziti-edge-controller:1280) version v0.29.0(3ca2dd2f4e7b 2023-07-13T15:53:37Z)  
> (14)[0.048] INFO ziti-sdk:ziti.c:1398 ziti\_set\_api\_session() ztx[0] api session set, setting api\_session\_timer to 1740s  
> (14)[0.048] INFO tunnel-cbs:ziti\_tunnel\_ctrl.c:726 on\_ziti\_event() ziti\_ctx[indocker] connected to controller  
> (14)[0.048] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1144 on\_event() ztx[/tmp/indocker.client.json] context event : status is OK  
> (14)[0.051] INFO ziti-sdk:channel.c:234 new\_ziti\_channel() ch[0] (ziti-edge-router@tls://ziti-edge-router:3022) new channel for ztx[0] identity[indocker]  
> (14)[0.051] INFO tunnel-cbs:ziti\_tunnel\_ctrl.c:798 on\_ziti\_event() ztx[indocker] added edge router ziti-edge-router@tls://ziti-edge-router:3022@ziti-edge-router  
> (14)[0.051] INFO ziti-sdk:channel.c:728 reconnect\_channel() ch[0] reconnecting NOW  
> (14)[0.086] INFO ziti-sdk:channel.c:626 hello\_reply\_cb() ch[0] connected. EdgeRouter version: v0.29.0|3ca2dd2f4e7b|2023-07-13T15:53:37Z|linux|amd64  
> (14)[0.086] INFO tunnel-cbs:ziti\_tunnel\_ctrl.c:802 on\_ziti\_event() ztx[indocker] router ziti-edge-router@tls://ziti-edge-router:3022 connected  
> (14)[1.048] INFO ziti-sdk:posture.c:204 ziti\_send\_posture\_data() ztx[0] first run or potential controller restart detected

The logs look good and I can see the identity connected in ZAC/UI

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 22, 2023, 2:26am UTC](https://openziti.discourse.group/t/published-tunnel-image-not-working-as-expected/1516/2 "2023-08-22T02:26:42Z")

</div>

Hey @bearrito!

I see that you were able to to test the tunneler in `run` mode as an intercepting proxy with `ziti-edge-tunnel` installed in your custom container image, but for some reason it didn’t work with the official image.

I have definitely used the Ziti tunneler running in this container image, both in `run` mode and `run-host` mode, for an intercepting proxy and hosting-only reverse proxy, respectively.

I think I’m seeing the same thing you reported. The tunneler connects to the controller, gets a list of routers, then a certificate verification fails when it connects to the first router.

```auto
handshake error error:0A000086:SSL routines::certificate verify failed

```

Here’s how I enabled the debug output:

```bash
docker compose run --env TLSUV_DEBUG=4 ziti-tun run --verbose=4

```

I’ll keep looking at precisely why that handshake failed. I didn’t encounter the controller TLS handshake failure you reported at all, so far it’s just the router in my test env.

I suspect the reason it worked in your custom container image is that you installed the binary from the DEB package, and that is built separately for each target OS.

We released the 0.22.5 DEB package today, and ~~I’m curious to learn if you have the same success with that version in your custom container image.~~ I triggered the same handshake failure with that version by installing the Ubuntu Jammy DEB.

To narrow down the variables it will be helpful to pin some container and binary versions so we know exactly what we’re dealing with.

To be cont’d…

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 22, 2023, 3:27am UTC](https://openziti.discourse.group/t/published-tunnel-image-not-working-as-expected/1516/3 "2023-08-22T03:27:40Z")

</div>

Here’s the Dockerfile I used to pin the version that worked for both of us.

```plaintext
FROM ubuntu:22.04
RUN apt-get update \
    && apt-get install -y netcat nano gpg curl \
    && curl -sSLf https://get.openziti.io/tun/package-repos.gpg \
        | gpg --dearmor --output /usr/share/keyrings/openziti.gpg \
    && chmod +r /usr/share/keyrings/openziti.gpg \
    && echo 'deb [signed-by=/usr/share/keyrings/openziti.gpg] https://packages.openziti.org/zitipax-openziti-deb-stable jammy main' \
        > /etc/apt/sources.list.d/openziti.list \
    && apt-get update \
    && apt-get install -y ziti-edge-tunnel=0.21.5 \
    && apt-mark hold ziti-edge-tunnel \
    && apt-get --yes autoremove \
    && apt-get clean autoclean \
    && rm -fr /var/lib/apt/lists/{apt,dpkg,cache,log} /tmp/* /var/tmp/*

```

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [August 24, 2023, 3:46pm UTC](https://openziti.discourse.group/t/published-tunnel-image-not-working-as-expected/1516/4 "2023-08-24T15:46:54Z")

</div>

@bearrito We found a problem with the quickstart version that was current at the time you wrote this topic, so I think the best way to proceed is to start fresh with the latest Compose quickstart version by setting in your `.env` file in the Compose project directory `ZITI_VERSION=0.30.0`. This var determines the `openziti/quickstart` container image tag in `docker-compose.yml`.