TLDR – there appears to be a bug in ziti-edge-tunneler new version release 0.20.0 this is preventing name resolution. Rolling back to 0.19.11 fixes the behavior immediately. I looked through all my networking config to try to determine root cause before realizing it’s a version related problem. I’ll leave this below in case it’s worth something, but the cmd/config outputs below are more or less identical on either version (with the exception of dig working on the last version but not the new one).
John
Debugging name resolution not working on ziti-edge-tunneler v0.20.0
Ok, back to the tunneler DNS setup and working on debugging around that…
So I’ve set up a simple service. Just replicating what was in one of the last ZitiTV video,
# ziti edge show config netcat-example -- BIND
{
"hostname": "localhost",
"port": 54321,
"protocol": "tcp"
}
# ziti edge show config netcat-example-client -- DIAL
{
"hostname": "ken.nc.ziti",
"port": 12345
}
The dial service is assigned by service policy to an identity on a x86-64 NixOS system with resolved enabled and 0.20.0 linux tunneler. I can see that when I start the tunneler up, the DNS functionality appears to be ok as indicated by the following log lines with trace debug enabled (snipping anything not DNS related):
...
(3248582)[ 0.000] DEBUG ziti-edge-tunnel:tun.c:295 init_dns_maintainer() setting up NETLINK listener
(3248582)[ 0.000] INFO tunnel-cbs:ziti_dns.c:171 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(3248582)[ 0.000] DEBUG tunnel-sdk:ziti_tunnel.c:321 ziti_tunneler_intercept() intercepting address[udp:100.64.0.2/32:53] service[ziti:dns-resolver]
(3248582)[ 0.014] DEBUG ziti-edge-tunnel:tun.c:271 on_dns_update_time() queuing DNS update
(3248582)[ 0.014] INFO ziti-edge-tunnel:resolvers.c:430 is_systemd_resolved_primary_resolver() Detected systemd-resolved is primary system resolver
(3248582)[ 0.014] INFO ziti-edge-tunnel:resolvers.c:66 init_libsystemd() Initializing libsystemd
(3248582)[ 0.014] DEBUG ziti-edge-tunnel:resolvers.c:93 init_libsystemd() Dynamically loaded libsystemd
(3248582)[ 0.014] DEBUG ziti-edge-tunnel:resolvers.c:329 try_libsystemd_resolver() Detected systemd is init system
(3248582)[ 0.014] DEBUG ziti-edge-tunnel:resolvers.c:336 try_libsystemd_resolver() Connected to system DBus
(3248582)[ 0.014] DEBUG ziti-edge-tunnel:resolvers.c:206 sd_bus_is_acquired_name() systemd-resolved DBus name found: org.freedesktop.resolve1
(3248582)[ 0.014] INFO ziti-edge-tunnel:resolvers.c:343 try_libsystemd_resolver() systemd-resolved selected as dns resolver manager
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:313 set_systemd_resolved_link_setting() Success in method invocation: SetLinkLLMNR for link: (tun0)
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:313 set_systemd_resolved_link_setting() Success in method invocation: SetLinkMulticastDNS for link: (tun0)
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:313 set_systemd_resolved_link_setting() Success in method invocation: SetLinkDNSOverTLS for link: (tun0)
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:313 set_systemd_resolved_link_setting() Success in method invocation: SetLinkDNSSEC for link: (tun0)
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:313 set_systemd_resolved_link_setting() Success in method invocation: SetLinkDNS for link: (tun0)
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:391 dns_update_systemd_resolved() Setting empty domain on interface: tun0
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:313 set_systemd_resolved_link_setting() Success in method invocation: SetLinkDomains for link: (tun0)
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:168 sd_bus_run_command() Success in command invocation: FlushCaches
(3248582)[ 0.016] DEBUG ziti-edge-tunnel:resolvers.c:168 sd_bus_run_command() Success in command invocation: ResetServerFeatures
(3248582)[ 0.018] DEBUG ziti-edge-tunnel:tun.c:266 after_set_dns() DNS update: 0
...
(3248582)[ 2.409] INFO tunnel-cbs:ziti_tunnel_cbs.c:403 new_ziti_intercept() creating intercept for service[netcat-example] with ziti-tunneler-client.v1 = {"hostname":"ken.nc.ziti","port":12345}
(3248582)[ 2.409] INFO tunnel-cbs:ziti_dns.c:297 new_ipv4_entry() registered DNS entry ken.nc.ziti -> 100.64.0.3
(3248582)[ 2.409] DEBUG tunnel-sdk:ziti_tunnel.c:321 ziti_tunneler_intercept() intercepting address[udp:100.64.0.3/32:12345] service[netcat-example]
(3248582)[ 2.409] DEBUG tunnel-sdk:ziti_tunnel.c:321 ziti_tunneler_intercept() intercepting address[tcp:100.64.0.3/32:12345] service[netcat-example]
...
(3248582)[ 55.986] TRACE tunnel-sdk:netif_shim.c:56 netif_shim_input() received packet UDP[100.64.0.1:35930 -> 100.64.0.2:53] len=88
(3248582)[ 55.986] TRACE tunnel-sdk:tunnel_udp.c:177 recv_udp() received datagram 100.64.0.1:35930->100.64.0.1:53
(3248582)[ 55.986] VERBOSE tunnel-sdk:intercept.c:113 address_match() ziti_address_cidr match score 0
(3248582)[ 55.986] VERBOSE tunnel-sdk:intercept.c:136 port_match() matching port 53 to range 12345
(3248582)[ 55.986] TRACE tunnel-sdk:tunnel_udp.c:201 recv_udp() no intercepted addresses match udp:100.64.0.2:53
(3248582)[ 55.986] TRACE tunnel-sdk:netif_shim.c:56 netif_shim_input() received packet UDP[100.64.0.1:36821 -> 100.64.0.2:53] len=88
(3248582)[ 55.986] TRACE tunnel-sdk:tunnel_udp.c:177 recv_udp() received datagram 100.64.0.1:36821->100.64.0.1:53
(3248582)[ 55.986] VERBOSE tunnel-sdk:intercept.c:113 address_match() ziti_address_cidr match score 0
(3248582)[ 55.986] VERBOSE tunnel-sdk:intercept.c:136 port_match() matching port 53 to range 12345
(3248582)[ 55.986] TRACE tunnel-sdk:tunnel_udp.c:201 recv_udp() no intercepted addresses match udp:100.64.0.2:53
Resolvctl and the resolver stub file seems to show that DNS intercepts from OpenZiti tunneler at 100.64.0.2 should be properly directed:
❯ cat /run/systemd/resolve/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
...
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 100.64.0.2
search .
❯ resolvectl status
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=allow-downgrade/unsupported
resolv.conf mode: stub
Current DNS Server: 8.8.8.8
DNS Servers: 8.8.8.8 8.8.4.4
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google
2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google
Link 3 (wlp4s0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
<...snip other interfaces...>
Link 267 (tun0)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.64.0.2
DNS Servers: 100.64.0.2
However, if I check interface and port bindings, I don’t see the tunneler directly binding tcp/udp port 53 on interface 100.64.0.2 in netstat. Shouldn’t we see it here?
❯ sudo netstat -tupln | grep 53
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 3245770/systemd-res
tcp 0 0 127.0.0.54:53 0.0.0.0:* LISTEN 3245770/systemd-res
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 3245770/systemd-res
tcp6 0 0 :::5355 :::* LISTEN 3245770/systemd-res
udp 0 0 127.0.0.54:53 0.0.0.0:* 3245770/systemd-res
udp 0 0 127.0.0.53:53 0.0.0.0:* 3245770/systemd-res
udp 0 0 0.0.0.0:5355 0.0.0.0:* 3245770/systemd-res
udp6 0 0 :::5355 :::* 3245770/systemd-res
IP information shows the tun0 device is set up properly for 100.64.0.1, nothing seen for 100.64.0.2. I’m thinking the 100.64.0.2 packets are handled indirectly though iptables rules or ip route instead of direct device and ip addr registration:
❯ ip a
<...snip...>
267: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
link/none
inet 100.64.0.1/32 scope global tun0
valid_lft forever preferred_lft forever
inet6 fe80::fc94:f4be:91fd:b0/64 scope link stable-privacy
valid_lft forever preferred_lft forever
iptables-save, iptables -t nat -v --list, iptables -t mangle -v --list all show no rules which would intercept packets on 100.64.0.2 – ie: no hit for | grep '100.' in the output of those commands. Looks like this is handled by ip route:
❯ ip route
default via 192.168.3.1 dev wlp4s0 proto dhcp src 192.168.3.92 metric 3003 mtu 1500
<...snip non-relevant routes...>
$PUBLIC_CONTROLLER_IP via 192.168.3.1 dev wlp4s0
100.64.0.0/10 dev tun0 scope link
100.64.0.2 dev tun0 scope link
100.64.0.3 dev tun0 scope link
But name resolution for the tunneler at 100.64.0.2 doesn’t appear to work yet:
# Should receive an answer
❯ dig ken.nc.ziti
; <<>> DiG 9.18.3 <<>> ken.nc.ziti
;; global options: +cmd
;; connection timed out; no servers could be reached
# Should receive an answer
❯ dig @100.64.0.2 ken.nc.ziti
; <<>> DiG 9.18.3 <<>> @100.64.0.2 ken.nc.ziti
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
# Expected
❯ dig @100.64.0.1 ken.nc.ziti
;; communications error to 100.64.0.1#53: connection refused
Oh! If I rollback to version 0.19.11, resolution works right away… Seems something has broken in 0.20.0.
