Ziti-tunnel.resolver.test

1

How do I resolve the error:

ERROR ziti/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup zitr.test on 127.0.0.11:53: no such host

Im using the docker-compose quickstart on Windows Subsystem for Linux where I have now added an additional webserver, nfs server and moodle containers to teh networks which appear to work - "most" of the time - but intermittently I get DNS issues where it cant resolve the ziti names for these services. I added in "prepend domain-name-servers 127.0.0.1:53;" to the dhclient.conf in the WSL Ubuntu server as suggested in the output but it made no real difference and the error just repeats after stopping and starting the docker containers:

ziti-edge-router-1 | [ 2.441] INFO ziti/tunnel/dns.NewDnsServer: dns server running at 127.0.0.1:53
ziti-edge-router-1 | [ 2.441] INFO ziti/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
ziti-edge-router-1 | [ 2.489] ERROR ziti/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup zitr.test on 127.0.0.11:53: no such host
ziti-edge-router-1 |
ziti-edge-router-1 | ziti-tunnel runs an internal DNS server which must be first in the host's
ziti-edge-router-1 | resolver configuration. On systems that use NetManager/dhclient, this can
ziti-edge-router-1 | be achieved by adding the following to /etc/dhcp/dhclient.conf:
ziti-edge-router-1 |
ziti-edge-router-1 | prepend domain-name-servers 127.0.0.1:53;
ziti-edge-router-1 |
ziti-edge-router-1 |
ziti-edge-router-1 | [ 2.489] INFO ziti/tunnel/intercept.SetDnsInterceptIpRange: dns intercept IP range: 100.64.0.1 - 100.127.255.255
ziti-edge-router-wss-1 | [ 2.507] INFO ziti/tunnel/dns.NewDnsServer: dns server running at 127.0.0.1:53
ziti-edge-router-wss-1 | [ 2.507] INFO ziti/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
ziti-edge-router-wss-1 | [ 2.515] ERROR ziti/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup zitr.test on 127.0.0.11:53: no such host
ziti-edge-router-wss-1 |
ziti-edge-router-wss-1 | ziti-tunnel runs an internal DNS server which must be first in the host's
ziti-edge-router-wss-1 | resolver configuration. On systems that use NetManager/dhclient, this can
ziti-edge-router-wss-1 | be achieved by adding the following to /etc/dhcp/dhclient.conf:
ziti-edge-router-wss-1 |
ziti-edge-router-wss-1 | prepend domain-name-servers 127.0.0.1:53;

Is there a way to fix this issue in WSL or am I missing something around this?

2

As amazing as it is, docker in general is "kind of a pain" in some ways. One of those ways is with DNS entries and resolving things inside the docker runtime and outside the runtime. It gets extra complex if your base OS is running a tunneler too (ziti-edge-tunnel, ziti desktop for windows/mac, etc)...

I've been personally experimenting with the tproxy/sidecar pattern with docker of late but found WSL's implementation to be "great for most things" -- but not for a lot of "networking things" and this is one of them. I myself had pains like this just this week, only to discover that it's a limitation in WSL itself and wasn't really "my" problem...

So that's a long-winded way of saying: "doing this stuff on one machine can be hard/painful". I ended up installing Oracle Virtual Box and Debian 12 so that I could have a "full-featured" linux to accomplish what I needed to.

To recap:

  • running a tunneler on your host OS -- causes issues because docker the TUN that's created is system-wide and runs the risk of confusing docker. You often WANT to do this though, because it's how you'll test.
  • using tproxy mode for routers in docker on windows just doesn't work with WSL-backed docker Are there plans to add the tproxy module? · Issue #10149 · microsoft/WSL · GitHub I tried to use docker desktop as well but ended up struggling with the difference between bash/wsl/cmd.exe so i made a VM and run it in bridged networking mode (so that my local tunneler doesn't impact it)

Can we maybe take a slightly different approach here? Can you explain the situation you're trying to understand and learn about and maybe I can make a suggestion as to what would work better?

3

Thanks for coming back to me .......I did originally start out with the docker compose quickstart on an Ubuntu VM which I had on a windows host which worked ok for the inital set up for me to play with. I moved it to WSL so that I could use the Windows Desktop Edge (that you use a lot in your videos) as I thought that would be easier for me to test things than using the Linux Tunneler that I was using on the VM.

If its better to use Ubuntu or Debian, Ideally, I would like to use the docker compose network that I have set up runnning on a VM and be able to access the network and ziti services I have created in it using the Windows Tunneler from my Host laptop - will that work based on what you said above ?

4

I'd tell you to use a VM at this point yeah, just because of the troubles I ended up hitting with WSL/docker and running a tunneler on the same machine as the docker network... So keep your docker compose based network running in docker on that VM, that's definitely what I'd do.

What you want to do will work just fine but there there is a minor challenge with that older compose environment. Mostly, the issue is that all/any participants in the overlay network will need to resolve the names of the containers. That's not a huge deal, but it might be something you've never done before. Basically, you either need to run a nameserver on your home network, or the cheap/easy way (but easy to forget about way) is to just use your hosts file (i almost always use my hosts file, but it's really easy to forget you added and entry).

Basically, your clients (sounds like it'd just be one tunneler anyway?) will need to be able to resolve ziti-edge-controller and ziti-edge-router as documented on the docker compose page: Local - Docker Compose | OpenZiti

For me, my C:\Windows\System32\drivers\etc\hosts file always has these two entries and I'll change that accordingly if I need to.

127.0.0.1 ziti-edge-controller ziti-edge-router

We've learned a lot since that original compose file was published years ago, there's newer stuff that @qrkourier has that you might find just as easy to use -- and perhaps easier. I expect the quickstarts to all get a fresh coat of paint later this year as we adapt to using the newer stuff.

So once you move the compose file to the VM and allow your hosting tunneler to resolve those two names, thing should work just fine. The intermittent nature of the issue also makes me think that maybe you have these entries already, but wsl doesn't add them back into the hosts file on restart of WSL?

hope that helps and makes sense...

5

I see this error even when the router is configured with host mode, so I ignore it unless I'm attempting to run the router in tproxy mode. I'll raise an issue to suppress that message when it's not meaningful.

Did you encounter a problem that led you to the error message in the container log, or were you investigating the error for its own sake?

Here's the instructions for a simplified Docker quickstart: ziti/quickstart/docker/all-in-one at main · openziti/ziti · GitHub

6

Thanks for the reply .....I have moved my docker-compose network to an Ubuntu VM running on a Windows Host - however the Tunneler running on the Windows Host is not able to enroll any identities from the docker compose network running on the VM

My host file on the windows host machine has these entries as suggested:

127.0.0.1 ziti-edge-controller ziti-edge-router

Is this an issue with the way the controller creates the certs in the docker compose setup on the VM ? - meaning the "outside" host machine is not able to resolve whats put in the identity file to enroll? Is there a way to fix this so the windows tunneler on my host windows machine can enroll identites created inside the docker compose environment on the Ubuntu VM ?

7

Yes, basically. When you make an identity, the JWT file that the identity will enroll with has the url of the controller baked INTO the jwt. so you need to make sure the advertised address in the configuration of the controller is what you expect... With the older docker compose quickstart, you'll have an advertised address in that JWT of "https://ziti-edge-controller:1280". That's how it comes out of the box.

That means to be able to successfully enroll, the windows client will need to access https://ziti-edge-controller:1280". You can see the advertised address from within the controller with:

$ docker compose exec ziti-controller grep 1280 ziti-controller.yaml
    address: ziti-edge-controller:1280
      - interface: 0.0.0.0:1280
        address: ziti-edge-controller:1280

So the question is now, can your windows machine access https://ziti-edge-controller:1280 ? If so - you should be able to create and enroll an identity

8

So for example, I just made an identity and copied it out to windows (i also have jq and base64 on my windows path)...

I can decode that jwt using a CLI (or jwt.io if you wish) and it will look like this:

C:\Users\clint>ziti edge create identity test -o test.jwt
New identity test created with id: 8qmcmE0iCA
Enrollment expires at 2024-08-15T23:27:42.805Z

C:\Users\clint>cut -d "." -f2 test.jwt | base64 -d | jq .
{
  "iss": "https://ziti-edge-controller:1280",
  "sub": "8qmcmE0iCA",
  "aud": [
    ""
  ],
  "exp": 1723764462,
  "jti": "c9ae2bd0-78cf-4965-bf13-39335309476f",
  "em": "ott",
  "ctrls": null
}

see the issuer? (iss) that field, your windows machine must be able to resolve: "https://ziti-edge-controller:1280"

And here's what it looks like in teh ZDEW after enrolling:
image

9

I went through what you described above and I had the same configuration but still could not connect

I went back to the host file on my Windows Host and added in the IP address of the Ubuntu VM (192.168.188.134 ziti-edge-controller) aswell like below :

127.0.0.1 ziti-edge-controller ziti-edge-router

192.168.188.134 ziti-edge-controller

On doing that I was then able to access the ziti console - I then enrolled the identity and it appeared like you have above

Apologies if that was a step you assumed I had in if it was meant to be there too ?

10

Ahhh, yeah. I forgot you were running it in a VM and that VM has a different IP address. You got it right... You should change your hosts file though. you probably want just one line:

192.168.188.134 ziti-edge-controller ziti-edge-router

The ziti-edge-router needs to be addressable too from your ZDEW or else traffic won't flow! :slight_smile:

You can check that advertised address with something like:

$ docker compose exec ziti-edge-router grep -B2 -A2 advertise ziti-edge-router.yaml
    - binding:          transport
      bind:             tls:0.0.0.0:10080
      advertise:        tls:ziti-edge-router:10080
      options:
        outQueueSize:   4
--
    address: tls:0.0.0.0:3022
    options:
      advertise: ziti-edge-router:3022
      connectTimeoutMs: 5000
      getSessionTimeout: 60

So this is showing you that other routers will try to connect to this router at: tls:ziti-edge-router:10080 and edge devices will try to connect to the edge (data) plane at ziti-edge-router:3022

So make sure your ZDEW can connect to that port too! We always recommend using/learinging openssl s_client -connect to test that port:

openssl s_client -connect ziti-edge-router:3022

make sure you see "stuff" coming back. For example at the end you should see:

Verify return code: 20 (unable to get local issuer certificate)
11

Im getting the same output as you have above when I run:

docker compose exec ziti-edge-router grep -B2 -A2 advertise ziti-edge-router.yaml

running the openssl check give me back "stuff" ending with:

ziti ssl
ziti ssl1368×197 28.5 KB

That is what is expected ?

12

Yes as long as that was from the windows machine. That means the windows machine can connect to the router's data port

13

Ah, no, that was the Ubuntu VM .... i'll install openssl on windows host and test again there :+1:

14

FYI, modern windows will have openssl on it already -- or it's a feature you can add through windows features.

For example:
C:\Windows\System32\openssl

15

I started this on a windows 10 home laptop that was lying around and is not installed ... anyway, i updated the hosts file to include the ziti-edge-router also pointing at the ubuntu vm address and went ahed and created test service and it displays so all appears working:

ziti-w
ziti-w982×381 12.5 KB

1 Like
16

Just noticed in the cli console there are these errors which keep repeating even though things appear to be working ?

tls error
tls error1849×478 146 KB

17

When you see those sorts of repeating errors every 10 to 15 seconds like that, it generally means there's a tunneler of some kind that is on but the identity is incorrect. Do you have a ziti-edge-tunnel or other ZDEW trying to connect somewhere? It might be the ZDEW too, if there's an older identity that's not working. Cycle the big green power button on the ZDEW and see if it stops. If there are older identities, remove them first, then cycle the on/off button on the ZDEW.

Looks like you got it working though, nice!