Docker:ziti-edge-tunnel - enrollment error (CONTROLLER_UNAVAILABLE)

Tunneler Apps Ziti Edge Tunnel
1

Hi Ziti folks
Greetings!

i have been following the video Install OpenZiti and Protect a Web Server Overview - YouTube

The setup is similar to the one in the video…

  1. ziti-overlay-network deployed on aws instance.

  2. As in the video a windows machine acts as the client.

    • ziti edge create identity user windowsweb -o windowsweb.jwt (on the aws instance)
    • Enrolled using the windows tunneller - Status says enabled.

  3. Created docker network

    • docker network create myFirstZitiNetwork

  4. The http-server is hosted in a docker container inside an ubuntu (20.04) machine.

    • docker run --rm --network myFirstZitiNetwork --network-alias sample-web-app crccheck/hello-world
    • Tested the server (as in the video)

  5. To run the docker command for ziti-edge-tunnel.

    • ziti edge create identity device ubuvm -o ubuvm.jwt
    • cp ubuvm.jwt ~/.ziti/ids
    • docker run --rm --network myFirstZitiNetwork --network-alias zet --name ziti-edge-tunnel --volume ~/.ziti/ids:/ziti-edge-tunnel --env=NF_REG_NAME=ubuvm openziti/ziti-edge-tunnel:latest run-host

    The enrollment fails :

INFO: identity file /ziti-edge-tunnel/ubuvm.json does not exist
INFO: looking for /var/run/secrets/netfoundry.io/enrollment-token/ubuvm.jwt
INFO: looking for /enrollment-token/ubuvm.jwt
INFO: looking for /ziti-edge-tunnel/ubuvm.jwt
INFO: enrolling /ziti-edge-tunnel/ubuvm.jwt
(8)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(8)[ 0.000] INFO ziti-sdk:utils.c:188 ziti_log_set_level() set log level: root=3/INFO
(8)[ 0.000] INFO ziti-sdk:ziti_enroll.c:92 ziti_enroll() Ziti C SDK version 0.32.6 @2fc3556(HEAD) starting enrollment at (2023-06-02T05:37:03.085)
(8)[ 0.515] ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[54.158.154.10] request failed: -103(software caused connection abort)
(8)[ 0.515] ERROR ziti-sdk:ziti_enroll.c:222 well_known_certs_cb() /__w/ziti-tunnel-sdk-c/ziti-tunnel-sdk-c/build/_deps/ziti-sdk-c-src/library/ziti_enroll.c:141 - ZITI_JWT_VERIFICATION_FAILED => -7 (JWT verification failed)
(8)[ 0.515] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:2116 enroll_cb() enrollment failed: CONTROLLER_UNAVAILABLE(-7)
ERROR: failed to enroll with token from /ziti-edge-tunnel/ubuvm.jwt (892B)

Not sure what i have missed. Pl let me know if any additional info is required.
It would be highly appreciated, the mistake is pointed out.

thanks in advance
santhosh

2

Hi @santvasu, welcome to the forum and OpenZiti!

Looking at your logs, it looks like the ziti-edge-tunnel container can’t contact the controller. That’s usually because the containers aren’t on the same network, but I can see from your command that you put the ziti-edge-tunnel onto the docker: myFirstZitiNetwork … so I don’t think that’s the case.

Let’s peek inside your jwt… can you run something like (assuming you have jq and base64, if not install them or exec into the controller because I know these tools are in that container):

cut -d "." -f2 /path/to/ubuvm.jwt | base64 -d | jq .

You should see something like this:

{
  "em": "erott",
  "exp": 1686313442,
  "iss": "https://ziti-edge-controller:1280",
  "jti": "4b668835-7fcb-4aeb-8c14-83eb5c5b8526",
  "sub": "-7fq.UJ7Ui"
}

The iss (issuer) field in ther will refer to the controller that created the identity. Can you make sure the ziti-edge-tunnel can access the url specified in there? I just started my own docker environment, so yours might have that exact url.

If the ziti-edge-tunnel container can curl/access that url, then something else is probably happening. It could also be that the jwt has expired. Notice the exp field? That represents the time the JWT will expire.

Let’s start there and see if we can figure out what’s going on.

3

Hi @TheLumberjack

Thank you very much for the quick reply.
Also thank you for the videos you have posted on openziti. They are very informative and instructive …

Back you the issue…
cut -d "." -f2 /path/to/ubuvm.jwt | base64 -d | jq .
{
“em”: “ott”,
“exp”: 1685694888,
“iss”: “https://54.158.154.10:8441”,
“jti”: “c4ac2b46-5ec2-491a-b639-bfbfbf073d7d”,
“sub”: “Qu0KM17H1J”
}

i would like to clarify one thing about the “ziti-edge-tunnel container and the controller” , being on the same network…

a) controller, edge-router are on the aws instance (54.158.154.10) (machine where expressInstall was run. (following the procedure “Host OpenZiti Anywhere”)

b) the ziti-edge-tunnel` container is on a ubuntu machine behind my home router.

c) the windows machine is also behind my home router. This runs the “Ziti Desktop Edge” GUI, and it was able to add the identity windowsweb.jwt

d) from my windows machine i was able to get the json output when pointing to
https://54.158.154.10:8441
{“data”:{“apiVersions”:{“edge”:{“v1”:{“apiBaseUrls”: /*snip … */ }

e) “exp” field, like i transfered the .jwt files fast enough (say max 5min), from the aws instance to the local machines (windows,ubuntu) machine.

f) i also tried installing the linux version of ziti-edge-tunnel and copied the identity file to /opt/openziti/etc/identities and restarted ziti-edge-tunnel. This also produced the same error messages (as in the docker case)

pl let me know if my description is clear enough. and if my understanding is correct.

thanks again

4

Oh my goodness, I’m sorry. After rereading the issue, I don’t know why I ended up thinking the ziti-edge-tunnel and overlay network were co-located in docker at all. You clearly stated that. My bad! Thanks for the clarification :slight_smile:

I think you found a bug in our quickstart!

I checked out the certificate presented by your controller and I think there’s a problem/bug with our quickstart. Looking at the certificate in my browser, going down to “Certificate Subject Alternative Name” I see you have: “DNS Name: 54.158.154.10” (twice) and IP Address: “54.158.154.10” (also twice):
image

I have definitely seen that situation cause problems in the past and to be honest, I’m surpised the Ziti Desktop Edge for Windows worked…

Steps to reproduce:

I suspect you ran the quickstart with EXTERNAL_DNS set to the address? Is that correct? We’ll have to try this out on our side… @gberl002 can you try setting EXTERNAL_DNS to the same IP and following the quickstart and see what you see when enrolling an identity in Mac/Linux (I can test Windows when you get it setup)

After we test and confirm that it’s a problem, the easiest thing for you to do is to tear things down and reprovision your overlay network. I know that’s a bit of a pain, but getting this PKI right is a really important step and as this is, it’s not right :frowning: Will that be a big deal?

We’ll try to reproduce the problem, thanks again for the report and stay tuned!

5

Hi @TheLumberjack
Pl don’t say “your bad”. I guess you would be looking into numerous posts and it quite likely that something in my initial post may not have been explicit, and has been overlooked. i am thankful to you for your detailed replies.

Yes: i ran the quickstart-Host OpenZiti Anywhere
with EXTERNAL_DNS="$(curl -s eth0.me)"

One doubt (may be it will look silly).
a) QuickStart-local-no-docker : This impiles that
  Ziti Overlay Network (controller,egde-router), + the zero-trust applications reside on the
   same physical machine (eg all in one aws instance)

b) Quickstart-Host OpenZiti Anywhere : This implies that
  Ziti Overlay Network (controller,egde-router) resides on one machine (say aws instance 1)
   Zero-trust application are hosted on other machine (say aws instance 2) or some machine behind a route (home/organization)

Meanwhile i will do the setup from scratch once again and confirm the same.

thanks again for the time and effort put into answering our queries

6

It appears this is a bug with the ziti tunneler. I haven’t figured out the exact details but I was able to reproduce the issue when using the IP for both EXTERNAL_IP and EXTERNAL_DNS. I’m still working on figuring out if it’s the duplicated SANs entry or something else but I wanted to provide an update nonetheless.

I’ll continue debugging this and get back to you with a more definitive answer.

7

thank you for reporting the issue. Fix is coming soon.

root cause issue: custom verification callback is not called for IP addressed servers (mbedtls) · Issue #162 · openziti/tlsuv · GitHub

2 Likes