Hi all,
Im having a hard time enrolling a docker instance of the edge tunneler. It seems that the image cannot contact the controller, but I have verified the connection in numerous ways. Any idea and assistance is highly appreciated. Thanks in advance!
Heres the error log from the image:
[root@docker ziti]# docker run --name ziti-host --rm --network=host --env ZITI_IDENTITY_BASENAME="somename" --env ZITI_IDENTITY_WAIT=10 --volume /root/ziti/somename.jwt:/ziti-edge-tunnel/somename.jwt openziti/ziti-host
WARN: the identities directory is only available inside this container because /ziti-edge-tunnel is not a mounted volume. Be careful to not publish this image with identity inside or lose access to the identity by removing the image prematurely.
INFO: setting NF_REG_NAME to ${ZITI_IDENTITY_BASENAME} (somename)
INFO: setting NF_REG_WAIT to ${ZITI_IDENTITY_WAIT} (10)
DEBUG: waiting 10s for /ziti-edge-tunnel/somename.json (or token) to appear
INFO: identity file /ziti-edge-tunnel/somename.json does not exist
INFO: looking for /var/run/secrets/netfoundry.io/enrollment-token/somename.jwt
INFO: looking for /enrollment-token/somename.jwt
INFO: looking for /ziti-edge-tunnel/somename.jwt
INFO: enrolling /ziti-edge-tunnel/somename.jwt
(8)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(8)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(8)[ 0.000] INFO ziti-sdk:ziti_enroll.c:92 ziti_enroll() Ziti C SDK version 0.33.4 @27bac90(HEAD) starting enrollment at (2023-08-24T15:05:10.441)
(8)[ 0.065] ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[controlleradress] request failed: -103(software caused connection abort)
(8)[ 0.065] ERROR ziti-sdk:ziti_enroll.c:235 enroll_cb() failed to enroll with controller: https://controlleradress:1280 CONTROLLER_UNAVAILABLE (software caused connection abort)
(8)[ 0.065] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2137 enroll_cb() enrollment failed: CONTROLLER_UNAVAILABLE(-3)
ERROR: failed to enroll with token from /ziti-edge-tunnel/somename.jwt (888B)
I see the ziti-host container uses the Docker host's host network mode, which is useful for the container to serve as a reverse-proxy to servers reachable by the Docker host.
For this to work, https://controlleradress:1280/ needs to be reachable from the Docker host. Can you confirm that is the case?
Hi @qrkourier,
thanks for your quick reply. Yes, the address is resolvable and reachable from the docker host. I have verified this via curl:
[root@docker ziti]# curl -k https://someaddress:1280
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://someaddress:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://someaddress:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://someaddress:1280/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2023-04-12T18:24:17Z","revision":"e4520fd2c0ed","runtimeVersion":"go1.20.2","version":"v0.27.9"},"meta":{}}
That the container uses the host network is actually already part of my debugging process. Initially I wanted to create a network shared by some of the containers on the host and the tunneler, to make the services available via ziti.
That cURL test shows a good result from the network perspective of the Docker host. That is strange the container is unable to either resolve the same name or reach the resolved IP address.
What result do you get with a busybox container that's using the same host network mode?
docker run --rm --network=host busybox wget -O- https://someaddress:1280/
Hi @qrkourier,
thanks for the hint. This is the output of the command you mentioned:
[root@docker ~]# docker run --rm --network=host busybox wget -O- https://someaddress:1280/
Connecting to someaddress:1280 (192.168.11.52:1280)
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer
I guess this proves that the controller is reachable. The failed certificate check was expected, right?
Addendum: This command yields the same output without the "--network=host" flag.
Good, so it's reachable from the busybox container. I forgot to add --no-check-certificate. You should see the JSON response.
I suppose Docker wouldn't let you create a bridge network named "host," so it must be using the Docker host's network. It works from a bridge too so that concern is moot.
Let's turn up the log level on the ziti-host container.
Set these command args: run-host --verbose=4 e.g.
docker run \
--name ziti-host \
--rm \
--network=host \
--env ZITI_IDENTITY_BASENAME="somename" \
--env ZITI_IDENTITY_WAIT=10 \
--volume /root/ziti/somename.jwt:/ziti-edge-tunnel/somename.jwt \
openziti/ziti-host \
run-host \
--verbose=4
Hi, @qrkourier,
there seems to be something strange with my controller. Adding the --no-check-certificate flag does not yield the JSON. The handshake still fails. I added the DNS option for debugging purposes.
Raising the debug level did not yield additional information.
Any help is appreciated.
[root@docker ~]# docker run --rm --network=host --dns=8.8.8.8 busybox wget --no-check-certificate -O- https://controller:1280
Connecting to controller:1280 (IPADDRESS:1280)
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer
I scanned the available cipher suites on the controller with nmap and found a warning. Might this be part of the issue?
[root@docker ~]# nmap --script ssl-enum-ciphers -p 1280 controller
Starting Nmap 7.70 ( https://nmap.org ) at 2023-08-28 16:42 CEST
Nmap scan report for controller (IPADDRESS)
Host is up (0.00057s latency).
PORT STATE SERVICE
1280/tcp open pictrography
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
| TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (secp256r1) - A
| compressors:
| NULL
| cipher preference: server
| warnings:
| Key exchange (secp256r1) of lower strength than certificate key
|_ least strength: A
Nmap done: 1 IP address (1 host up) scanned in 1.08 seconds
From outside of the container, wget receives the JSON just fine:
[root@docker ~]# wget -nv --no-check-certificate -O- https://controller:1280
WARNING: The certificate of ‘controller’ is not trusted.
WARNING: The certificate of ‘controller’ hasn't got a known issuer.
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://controller:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://controller:1280/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://controller:1280/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2023-04-12T18:24:17Z","revision":"e4520fd2c0ed","runtimeVersion":"go1.20.2","version":"v0.27.9"},"meta":{}}
2023-08-28 16:46:01 URL:https://controller:1280/ [469/469] -> "-" [1]
To isolate the cause, will you perform the same docker run with the IP address of the controller instead of the domain name?
Also, do you see how to add the run-host --verbose=4 command args to turn-up the log level on the ziti-host container? Maybe that will tell us more about the problem from Ziti's perspective.
It appears that containers with network mode "host" can't resolve the controller's domain name in DNS.
No, I don't believe so. That appears to be an informational message from Nmap warning the user that the key exchange algorithm's elliptic curve has fewer bits than the curve used by the certificate, the implications of which are beyond my immediate grasp.
Thanks @qrkourier,
Having a dedicated network solved the enrollment problem. Now I have a different issue. The docker container seems to be unable to open up the tun device. Im using Alma Linux 8, but SELINUX is temporarily disabled.
[root@docker ~]# docker run --name ziti-host --rm --network=ziti-host-net --env ZITI_IDENTITY_BASENAME="controller" --env ZITI_IDENTITY_WAIT=10 --volume /root/ziti/controller.jwt:/ziti-edge-tunnel/controller.jwt --volume /root/ziti:/ziti-edge-tunnel --cap-add=NET_ADMIN --privileged openziti/ziti-host --verbose=4
INFO: setting NF_REG_NAME to ${ZITI_IDENTITY_BASENAME} (controller)
INFO: setting NF_REG_WAIT to ${ZITI_IDENTITY_WAIT} (10)
WARN: the identities directory is only available inside this container because /ziti-edge-tunnel is not a mounted volume. Be careful to not publish this image with identity inside or lose access to the identity by removing the image prematurely.
DEBUG: waiting 10s for /ziti-edge-tunnel/controller.json (or token) to appear
INFO: found identity file /ziti-edge-tunnel/controller.json
DEBUG: evaluating positionals: --verbose=4
INFO: running ziti-edge-tunnel
(8)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=4/DEBUG
(8)[ 0.000] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1542 run_tunnel() failed to open network interface: failed to open tun device:Operation not permitted
About to run tunnel service... ziti-edge-tunnel/docker-entrypoint.sh: line 25: kill: (8) - No such process
Oh, if you're still wanting to use this ziti-host container as a reverse-proxy only you may drop the --cap-add=NET_ADMIN --privileged options. Instead of --verbose=4 you need run-host --verbose=4.
For example:
docker run \
--name ziti-host \
--rm \
--network=ziti-host-net \
--env ZITI_IDENTITY_BASENAME="controller" \
--env ZITI_IDENTITY_WAIT=10 \
--volume /root/ziti/controller.jwt:/ziti-edge-tunnel/controller.jwt \
openziti/ziti-host \
run-host \
--verbose=4
Thank you @qrkourier ,
thats the missing piece. adding run-host works.
I wonder why this is not described in Containers | OpenZiti.
Excellent. We could add a troubleshooting heading under the hosting use case (reverse proxy). That'd be a good place for how to obtain the debug log.
Proposed: document verbose container args by qrkourier · Pull Request #723 · openziti/ziti-tunnel-sdk-c · GitHub