Ok, I will get those for you now…
Is there a way to update the client tokens from the ziti cli or delete all previous entries (identities, configs) so I don’t have to manually look through and delete/remake the identities/services/etc.?
I’m not sure what you’re looking for, but if you want to ‘delete all the things’ and do it ‘in bulk’ the easiest thing to do is use a ‘where true’ clause.
ziti edge delete service where true
ziti edge delete service-policy where true
ziti edge delete config where true
If you do that, I don’t recommend you delete the identity, but you could. But if you were going to go this path, I think it is just as easy to just start everything fresh… Stop the ziti services (ziti-controller/ziti-router) and delete the $HOME/.ziti folder that the quickstart generates and rerun the quickstart.
Ah I see, thanks for the info. I ask because I find myself needing to re-enroll tokens when the server and client VMs restart and I get an error saying the tokens are invalid. Getting the logs now…
If you're finding this, that's a problem I'm concerned about. I don't know why you'd need to re-enroll your identities unless the hostnames are changing? This is a separate problem we should probably get to the bottom of too, this sounds like it has the opportunity to cause other sorts of problems.
So I’m not sure what changed, before starting the tunnelers I went ahead and deleted all the configs/services/identities and remade them - I can curl 100.64.0.3. I’m still unable to resolve http.ziti though. I can still paste the logs if you want, they got a bit long lol
This is probably user error on my end then, I wasn't sure if I could continue using those tokens a day after creating them (I think I recall seeing they only lasted for about a day) so I just try to re-enroll them.
The jwt/enrollment token is short lived by default. After enrolled, the identities are good ‘basically forever’ so yeah you can reuse them.
Would you email the logs to clint at openziti.org? I am interested in seeing all those logs. Specifically any errors that might be in there.
It sounds like you were able to curl to 100.64.0.3 this time though and get the docker whale/http response? If so - that’s encouraging, right? 
Now we just need to sort out the DNS issue.
Can you do a nslookup http.ziti 100.64.0.2 and resolve it?
So, that is good news… that means your ziti is working. So, if you ping http.ziti you should get the same address “100.64.0.3”
I’ll share your logs with @JamminSoleng too, once you send them over.
I get the same symptoms with latest tunneler 0.20.9 from the bionic repo running on bionic: the Ziti nameserver is available on 100.64.0.2, the systemd-resolved stub resolver is available on 127.0.0.53, but systemd-resolved isn’t configured to send queries to the Ziti nameserver.
vagrant@ubuntu1804:~$ /opt/openziti/bin/ziti-edge-tunnel version
v0.20.9-local
vagrant@ubuntu1804:~$ sudo ss -lnup|grep 53
UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=678,fd=12))
vagrant@ubuntu1804:~$ pgrep -f ziti-edge-tunnel|xargs ps -fww
UID PID PPID C STIME TTY STAT TIME CMD
root 2945 1 0 01:45 ? Ssl 0:00 /opt/openziti/bin/ziti-edge-tunnel run --verbose=2 --dns-ip-range=100.64.0.1/10 --identity-dir=/opt/openziti/etc/identities
vagrant@ubuntu1804:~$ dig +short httpbin.ziti @100.64.0.2
100.64.0.5
vagrant@ubuntu1804:~$ curl 100.64.0.5/ip
{
"origin": "172.17.0.1"
}
There’s only one ERROR emitted by the tunneler during startup that’s related to systemd-resolved:
(3309)[ 0.017] ERROR ziti-edge-tunnel:resolvers.c:313 set_systemd_resolved_link_setting() Failure in method invocation: SetLinkDNSOverTLS for link: (tun0): (org.freedesktop.DBus.Error.UnknownMethod, Unknown method 'SetLinkDNSOverTLS' or interface 'org.freedesktop.resolve1.Manager'.)
The tunneler will keep running even if something goes wrong with DNS auto-configuration, and there are a lot of ways it can go wrong. Until we figure out exactly what’s sideways on Bionic you can add an installation step for Bionic Beaver boxen to configure Ziti DNS.
Here’s how I did it for my Bionic VM:
# /etc/systemd/resolved.conf
[Resolve]
DNS=100.64.0.2 4.2.2.1 4.2.2.2
FallbackDNS=
Domains=
#LLMNR=no
#MulticastDNS=no
DNSSEC=yes
Cache=yes
DNSStubListener=yes
vagrant@ubuntu1804:~$ sudo systemctl restart systemd-resolved
vagrant@ubuntu1804:~$ dig +short httpbin.ziti
100.64.0.5
vagrant@ubuntu1804:~$ curl httpbin.ziti/ip
{
"origin": "172.17.0.1"
}
1 Like
This also worked around the problem.
vagrant@ubuntu1804:~$ sudo systemd-resolve --set-dns=100.64.0.2 --set-domain=~. --set-dnssec=no --interface tun0
vagrant@ubuntu1804:~$ sudo systemctl restart systemd-resolved
vagrant@ubuntu1804:~$ dig +short httpbin.ziti
100.64.0.5
1 Like
I’ve tested this on Jammy and DNS does resolve as expected. Going back to @qrkourier post, we had also noticed that the SetLinkDNSOverTLS method was missing from the systemd-resolved service in Bionic, at least a out of the box install.
Thanks all for the help.
1 Like
Released the fix for Bionic today. Thanks @sabedevops for the patch.
1 Like
I want to ask regarding this one. I can curl the web.ziti from the client side. But when I tried to nslookup, it returned the non-existent domain. My environment uses the Host It Anywhere platform in the cloud. Is it because it's not in the local environment?
I would think this is due to nslookup. DNS-related tools often don't work like other tools, they specifically ignore certain DNS settings. Also, every operating system has small nuances which might cause slightly different behavior. On windows, for example, nslookup doesn't use the NRPT and thus when you use windows and try to perform an nslookup you must add the name server to the command.
Try
nslookup private.web 100.64.0.2
It should work perfectly fine once you append the name server. Alternatively, you could try PowerShell and resolve-dnsname.
1 Like