I’m figuring out how to Ziti in general and how to tunnel to a DNS service on Linux in particular. I’m finding there are some caveats, possibly due to some kind of conflict with the built in resolver in Ziti tunneler, and I’m not sure how to proceed with troubleshooting.
I had the tunneler running just fine after modifying the Network Manager connection to have the loopback IP for the resolver as the first nameserver (127.0.0.111), but after doing a Ctrl-C to return to the terminal I’m unable to run it again. Each attempt fails with
FATAL ziti/tunnel/dns.NewDnsServer: system resolver test failed
I’ve looked for stray routes and IPtables mangle rules and tun interfaces that might have somehow been left behind by the prior run when it was working. There is an empty chain named NF-INTERCEPT that is the target of all traffic in PREROUTING. I tried flushing PREROUTING but the result is the same.
❯ sudo iptables -tmangle -nvL PREROUTING
Chain PREROUTING (policy ACCEPT 80160 packets, 67M bytes)
pkts bytes target prot opt in out source destination
80160 67M NF-INTERCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
❯ sudo iptables -tmangle -nvL NF-INTERCEPT
Chain NF-INTERCEPT (1 references)
pkts bytes target prot opt in out source destination
As for the local resolver configuration, it was working when the NM connection profile was set to ignore DHCP’s suggested nameservers and I had prepended the tunneler’s resolver address to the NM connection profile’s list of nameservers. While troubleshooting this failing built-in resolver test that runs on tunneler startup I’ve also tried the ziti-tunnel recommended method of adding a line to dhclient.conf file like
prepend domain-name-servers 127.0.0.111:53;
and reverting the NM connection profile to obey DHCP’s recommended nameservers. That seems to be syntactically incorrect
❯ sudo dhclient -1 -d wlp3s0
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
/etc/dhcp/dhclient.conf line 25: semicolon expected.
prepend domain-name-servers 127.0.0.111:
^
/etc/dhcp/dhclient.conf line 25: expecting a statement.
prepend domain-name-servers 127.0.0.111:53;
^
/etc/dhcp/dhclient.conf line 27: semicolon expected.
^
Changing the line in dhclient.conf to omit the trailing port that was recommended by the ziti-tunnel output like
prepend domain-name-servers 127.0.0.111;
seems to have allowed dhclient to run without error. The output mentions this file which has the expected nameservers and order.
❯ cat /run/systemd/resolved.conf.d/isc-dhcp-v4-wlp3s0.conf
[Resolve]
DNS=127.0.0.111 8.8.8.8 208.67.220.220
❯ dhclient --version
isc-dhclient-4.4.1
❯ ll /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jul 18 19:39 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
…tells me I’m using systemd-resolved according to the Linux wiki in Ziti docs, but it’s not clear whether that means I’m not using dhclient which is clearly installed.
❯ echo -e "[Resolve]\nDNS=127.0.0.111" | sudo tee /etc/systemd/resolved.conf.d/ziti-tunnel.conf
tee: /etc/systemd/resolved.conf.d/ziti-tunnel.conf: No such file or directory
[Resolve]
DNS=127.0.0.111
❯ sudo mkdir -pv /etc/systemd/resolved.conf.d/
mkdir: created directory '/etc/systemd/resolved.conf.d/'
❯ echo -e "[Resolve]\nDNS=127.0.0.111" | sudo tee /etc/systemd/resolved.conf.d/ziti-tunnel.conf
[Resolve]
DNS=127.0.0.111
❯ sudo systemctl restart systemd-resolved
❯ sudo ~/Downloads/ziti-tunnel-linux/ziti-tunnel run --resolver udp://127.0.0.111:53 --identity ~/Downloads/kentestId.json
[ 0.000] INFO ziti/tunnel/intercept/tproxy.New: tproxy listening on tcp:127.0.0.1:33609
[ 0.000] INFO ziti/tunnel/intercept/tproxy.New: tproxy listening on udp:127.0.0.1:41156, remoteAddr: <nil>
[ 0.005] INFO ziti/tunnel/cmd/ziti-tunnel/subcmd.run: using tproxy interceptor
[ 0.007] INFO ziti/tunnel/dns.NewDnsServer: starting dns server...
[ 2.007] INFO ziti/tunnel/dns.NewDnsServer: dns server running at 127.0.0.111:53
[ 2.007] INFO ziti/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
[ 2.209] FATAL ziti/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test: no such host
ziti-tunnel runs an internal DNS server which must be first in the host's
resolver configuration. On systems that use NetManager/dhclient, this can
be achieved by adding the following to /etc/dhcp/dhclient.conf:
prepend domain-name-servers 127.0.0.111:53;
The last piece was to circumvent the resolved’s stub listener (127.0.0.53) and symlink libc’s /etc/resolv.conf to the dynamically-rendered resolv.conf provided by systemd-resolvd (resolv.conf instead of stub-resolv.conf).
❯ sudo ln -sfvn /run/systemd/resolve/resolv.conf /etc/resolv.conf
'/etc/resolv.conf' -> '/run/systemd/resolve/resolv.conf'
While I’m happy that this allows DNS intercept to function normally, it’s not yet clear what tradeoffs I’m making bypassing the built-in stub listener.
Also, this is the only remaining caveat of which I’m aware: bypassing the stub listener may cripple resolved in some way. The prior concern was allayed about working around the default resolver loopback listener address in ziti-tunnel with the --resolver param. It was found to be unnecessary in the latest version.
SYSTEMD-RESOLVED.SERVICE(8) indicates that apps should use glibc NSS or bus APIs instead of the stub listener whenever possible because not all capabilities can be mapped to the unicast DNS protocol i.e. stub listener e.g. link-local addressing, LLMNR Unicode domains. It is also clear that caching and validating DNS/DNSSEC are features of the now-circumvented stub listener.
There was one additional obstacle I needed to overcome becaues I’d manually run dhclient which had inserted interface-specific resolver files in /run/systemd/resolved.conf.d/. This caused DHCP’s recommended nameservers to have a higher precedence than the Ziti resolver that I’d specified in the resold config include in /etc/systemd/resolved.conf.d/. Deleting the DNS= directives that were inserted by dhclient solved this particular problem.