restarted the router and it errors out with following message -
[ 2.650] INFO edge/tunnel/dns.NewDnsServer: dns server running at 10.10.10.10:53
[ 2.650] INFO edge/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
[ 2.652] FATAL edge/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test on 127.0.0.53:53: no such host
ziti-tunnel runs an internal DNS server which must be first in the host's
resolver configuration. On systems that use NetManager/dhclient, this can
be achieved by adding the following to /etc/dhcp/dhclient.conf:
prepend domain-name-servers 10.10.10.10:53;
I tried doing that ( editing /etc/dhcp/dhclient.conf ) but it didnt help.
Looks like the self test is using 127.0.0.53 but the message told you to bind on 10.10.10.10. I wonder if there’s already a resolver on that loopback address. Try using 127.0.0.53 in the config and see what happens? If it doesn’t work can you give me the is version you’re using?
the demo was setup on VMs running ubuntu 22.04. Ubuntu uses systems-resolved for name resolution
If you are running with similar setup you may have missed these two steps:
I did that part. And its like chicken-egg situation. If I add that in /etc/systemd/resolved.conf then edge router does not come up ( dns resolution fails )
and if I remove it, edge router prints the message as above.
I am doing this on RHEL8 with systemd-resolved as well.
Yes I did and to make sure edge-router comes up I had to add my local DNS servers as subsequent entries so what I have in /etc/systemd/resolved.conf is -
here is output from newly installed ubuntu 22.04 server with lan ip 10.250.50.78 before and after setting DNS in resolved.conf. As you can see we see the issue that you have prior to setting the DNS= and restarting systemd-resolved but works as expected after.
ziggy@testbox:~$ ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:0c:29:5d:c7:2a brd ff:ff:ff:ff:ff:ff
altname enp2s1
inet 10.250.50.78/24 metric 100 brd 10.250.50.255 scope global dynamic ens33
valid_lft 84199sec preferred_lft 84199sec
inet6 fe80::20c:29ff:fe5d:c72a/64 scope link
valid_lft forever preferred_lft forever
ziggy@testbox:~$
ziggy@testbox:~$
ziggy@testbox:~$ resolvectl
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (ens33)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.250.50.254
DNS Servers: 10.250.50.254
ziggy@testbox:~$
ziggy@testbox:~$
ziggy@testbox:~$ cat /etc/systemd/resolved.conf
An update with v0.26.5 is available for ziti-router v0.25.10 from
[ 0.324] INFO ziti/ziti-router/subcmd.run: {configFile=[config.yml] routerId=[P5xIkiIbE6] build-date=[2022-05-24T19:19:48Z] revision=[a32a117f9472] os=[linux] arch=[amd64] version=[v0.25.10] go-version=[go1.18.2]} starting ziti-router
[ 0.324] WARNING edge/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[ 0.324] INFO fabric/router/forwarder.(*Faulter).run: started
[ 0.325] INFO fabric/router/forwarder.(*Scanner).run: started
[ 0.325] INFO fabric/router.(*Router).showOptions: ctrl = {“OutQueueSize”:4,“MaxQueuedConnects”:1,“MaxOutstandingConnects”:16,“ConnectTimeout”:1000000000,“DelayRxStart”:false,“WriteTimeout”:0}
[ 0.325] INFO fabric/router.(*Router).showOptions: metrics = {“ReportInterval”:60000000000,“MessageQueueSize”:10}
[ 0.325] INFO fabric/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[ 0.325] INFO fabric/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[ 0.325] INFO edge/router/xgress_edge.(*listener).Listen: {address=[tls:0.0.0.0:443]} starting channel listener
[ 0.325] INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[100] idleTime=[10s] poolType=[pool.listener.xgress_edge] minWorkers=[1] maxQueueSize=[50]} starting goroutine pool
[ 0.325] INFO fabric/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:0.0.0.0:443]
[ 0.325] INFO fabric/router.(*Router).startXgressListeners: created xgress listener [tunnel] at
[ 0.326] INFO edge/router/xgress_edge.(*Acceptor).Run: starting
[ 0.558] INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[0] maxWorkers=[32] maxQueueSize=[1000] idleTime=[30s] poolType=[pool.link.dialer]} starting goroutine pool
[ 0.558] INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {maxWorkers=[128] maxQueueSize=[1000] poolType=[pool.route.handler] idleTime=[30s] minWorkers=[0]} starting goroutine pool
[ 0.558] INFO edge/router/fabric.(*StateManagerImpl).StartHeartbeat: heartbeat starting
[ 0.558] INFO edge/router/xgress_edge.(*CertExpirationChecker).Run: waiting 8591h42m56.432572551s to renew certificates
[ 0.559] WARNING edge/tunnel/dns.flushDnsCaches: {error=[exec: “systemd-resolve”: executable file not found in $PATH]} unable to find systemd-resolve in path, consider adding a dns flush to your restart process
[ 0.559] INFO edge/tunnel/dns.NewDnsServer: starting dns server…
[ 0.560] INFO edge/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1: received server hello, replying
[ 0.630] INFO edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).instantSync: {strategy=[instant]} first api session syncId [cl7s4i676x4uiemh017ju9ie2], starting
[ 0.630] INFO edge/router/handler_edge_ctrl.(*apiSessionSyncTracker).Add: received api session sync chunk 0, isLast=true
[ 0.809] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {linkProtocol=[tls] routerVersion=[v0.25.13] linkId=[4WMo3WUGRhvdEY60KjKyIP] routerId=[pmBQvi6oE6] address=[tls:152.67.235.29:80]} dialing link
[ 0.809] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {routerVersion=[v0.25.13] linkId=[7SSEoeL3dfIp5xpv6oCFnx] routerId=[cuKiosjbE6] address=[tls:150.230.46.39:80] linkProtocol=[tls]} dialing link
[ 1.042] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {routerId=[pmBQvi6oE6] routerVersion=[v0.25.13] linkId=[4WMo3WUGRhvdEY60KjKyIP]} link destination support heartbeats
[ 1.058] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[7SSEoeL3dfIp5xpv6oCFnx] routerId=[cuKiosjbE6] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.272] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {routerVersion=[v0.25.13] linkId=[4WMo3WUGRhvdEY60KjKyIP] routerId=[pmBQvi6oE6]} link destination support heartbeats
[ 1.272] INFO fabric/router.(*xlinkAccepter).Accept: accepted new link [l/4WMo3WUGRhvdEY60KjKyIP]
[ 1.272] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {routerVersion=[v0.25.13] linkId=[4WMo3WUGRhvdEY60KjKyIP] routerId=[pmBQvi6oE6] address=[tls:152.67.235.29:80] linkProtocol=[tls]} link registered
[ 1.272] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {address=[tls:162354cb-06b0-4953-bc2a-1b10267ad4f4.production.netfoundry.io:6262] linkProtocol=[tls] routerVersion=[v0.25.13] linkId=[5YE7PZL8kHQXS27K47ypF0] routerId=[5xnOn.aTq]} dialing link
[ 1.301] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[7SSEoeL3dfIp5xpv6oCFnx] routerId=[cuKiosjbE6] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.301] INFO fabric/router.(*xlinkAccepter).Accept: accepted new link [l/7SSEoeL3dfIp5xpv6oCFnx]
[ 1.301] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {linkProtocol=[tls] routerVersion=[v0.25.13] linkId=[7SSEoeL3dfIp5xpv6oCFnx] routerId=[cuKiosjbE6] address=[tls:150.230.46.39:80]} link registered
[ 1.508] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[5YE7PZL8kHQXS27K47ypF0] routerId=[5xnOn.aTq] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.631] INFO edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).applySync: finished sychronizing api sessions [count: 5, syncId: cl7s4i676x4uiemh017ju9ie2, duration: 121.999µs]
[ 1.760] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[5YE7PZL8kHQXS27K47ypF0] routerId=[5xnOn.aTq] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.760] INFO fabric/router.(*xlinkAccepter).Accept: accepted new link [l/5YE7PZL8kHQXS27K47ypF0]
[ 1.760] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {address=[tls:162354cb-06b0-4953-bc2a-1b10267ad4f4.production.netfoundry.io:6262] linkProtocol=[tls] routerVersion=[v0.25.13] linkId=[5YE7PZL8kHQXS27K47ypF0] routerId=[5xnOn.aTq]} link registered
[ 2.560] INFO edge/tunnel/dns.NewDnsServer: dns server running at 10.250.50.78:53
[ 2.560] INFO edge/tunnel/dns.(*resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver [ 2.609] FATAL edge/tunnel/dns.NewDnsServer: system resolver test failed: failed to resolve ziti-tunnel.resolver.test: lookup ziti-tunnel.resolver.test: no such host
ziti-tunnel runs an internal DNS server which must be first in the host’s resolver configuration. On systems that use NetManager/dhclient, this can be achieved by adding the following to /etc/dhcp/dhclient.conf:
Link 2 (ens33)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
DNS Servers: 10.250.50.254
ziggy@testbox:~$
ziggy@testbox:~$
ziggy@testbox:~$ sudo ./ziti-router run config.yml
An update with v0.26.5 is available for ziti-router v0.25.10 from
[ 0.311] INFO ziti/ziti-router/subcmd.run: {go-version=[go1.18.2] configFile=[config.yml] revision=[a32a117f9472] os=[linux] arch=[amd64] routerId=[P5xIkiIbE6] build-date=[2022-05-24T19:19:48Z] version=[v0.25.10]} starting ziti-router
[ 0.311] WARNING edge/router/internal/edgerouter.(*Config).LoadConfigFromMap: Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]
[ 0.311] INFO fabric/router.(*Router).showOptions: ctrl = {“OutQueueSize”:4,“MaxQueuedConnects”:1,“MaxOutstandingConnects”:16,“ConnectTimeout”:1000000000,“DelayRxStart”:false,“WriteTimeout”:0}
[ 0.311] INFO fabric/router.(*Router).showOptions: metrics = {“ReportInterval”:60000000000,“MessageQueueSize”:10}
[ 0.311] INFO fabric/router.(*Router).initializeHealthChecks: starting health check with ctrl ping initially after 15s, then every 30s, timing out after 15s
[ 0.311] INFO fabric/router.(*Router).startXlinkDialers: started Xlink dialer with binding [transport]
[ 0.311] INFO edge/router/xgress_edge.(*listener).Listen: {address=[tls:0.0.0.0:443]} starting channel listener
[ 0.311] INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {minWorkers=[1] maxWorkers=[100] idleTime=[10s] maxQueueSize=[50] poolType=[pool.listener.xgress_edge]} starting goroutine pool
[ 0.311] INFO fabric/router.(*Router).startXgressListeners: created xgress listener [edge] at [tls:0.0.0.0:443]
[ 0.311] INFO fabric/router.(*Router).startXgressListeners: created xgress listener [tunnel] at
[ 0.312] INFO fabric/router/forwarder.(*Faulter).run: started
[ 0.312] INFO fabric/router/forwarder.(*Scanner).run: started
[ 0.312] INFO edge/router/xgress_edge.(*Acceptor).Run: starting
[ 0.582] INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[30s] poolType=[pool.link.dialer] maxQueueSize=[1000] minWorkers=[0] maxWorkers=[32]} starting goroutine pool
[ 0.582] INFO fabric/metrics.GoroutinesPoolMetricsConfigF.func1.1: {idleTime=[30s] poolType=[pool.route.handler] minWorkers=[0] maxQueueSize=[1000] maxWorkers=[128]} starting goroutine pool
[ 0.582] INFO edge/router/fabric.(*StateManagerImpl).StartHeartbeat: heartbeat starting
[ 0.582] INFO edge/router/xgress_edge.(*CertExpirationChecker).Run: waiting 8591h39m16.736026379s to renew certificates
[ 0.585] INFO edge/router/handler_edge_ctrl.(*helloHandler).HandleReceive.func1: received server hello, replying
[ 0.586] WARNING edge/tunnel/dns.flushDnsCaches: {error=[exec: “systemd-resolve”: executable file not found in $PATH]} unable to find systemd-resolve in path, consider adding a dns flush to your restart process
[ 0.586] INFO edge/tunnel/dns.NewDnsServer: starting dns server…
[ 0.657] INFO edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).instantSync: {strategy=[instant]} first api session syncId [cl7s4mvpvx50lemh05unarpr5], starting
[ 0.658] INFO edge/router/handler_edge_ctrl.(*apiSessionSyncTracker).Add: received api session sync chunk 0, isLast=true
[ 0.833] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {linkProtocol=[tls] routerVersion=[v0.25.13] linkId=[70Qlj2VeYhzLiODY2pwiZb] routerId=[pmBQvi6oE6] address=[tls:152.67.235.29:80]} dialing link
[ 0.834] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {routerVersion=[v0.25.13] linkId=[4hHaFVrii0Ye5rDnDrmdYW] routerId=[cuKiosjbE6] address=[tls:150.230.46.39:80] linkProtocol=[tls]} dialing link
[ 1.067] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[4hHaFVrii0Ye5rDnDrmdYW] routerId=[cuKiosjbE6] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.076] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[70Qlj2VeYhzLiODY2pwiZb] routerId=[pmBQvi6oE6] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.295] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {routerVersion=[v0.25.13] linkId=[4hHaFVrii0Ye5rDnDrmdYW] routerId=[cuKiosjbE6]} link destination support heartbeats
[ 1.295] INFO fabric/router.(*xlinkAccepter).Accept: accepted new link [l/4hHaFVrii0Ye5rDnDrmdYW]
[ 1.295] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {routerVersion=[v0.25.13] linkId=[4hHaFVrii0Ye5rDnDrmdYW] routerId=[cuKiosjbE6] address=[tls:150.230.46.39:80] linkProtocol=[tls]} link registered
[ 1.295] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {linkId=[5q2jdteIEC7dEMJO07qBu5] routerId=[5xnOn.aTq] address=[tls:162354cb-06b0-4953-bc2a-1b10267ad4f4.production.netfoundry.io:6262] linkProtocol=[tls] routerVersion=[v0.25.13]} dialing link
[ 1.300] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[70Qlj2VeYhzLiODY2pwiZb] routerId=[pmBQvi6oE6] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.300] INFO fabric/router.(*xlinkAccepter).Accept: accepted new link [l/70Qlj2VeYhzLiODY2pwiZb]
[ 1.300] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {address=[tls:152.67.235.29:80] linkProtocol=[tls] routerVersion=[v0.25.13] linkId=[70Qlj2VeYhzLiODY2pwiZb] routerId=[pmBQvi6oE6]} link registered
[ 1.532] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[5q2jdteIEC7dEMJO07qBu5] routerId=[5xnOn.aTq] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.658] INFO edge/router/handler_edge_ctrl.(*apiSessionAddedHandler).applySync: finished sychronizing api sessions [count: 5, syncId: cl7s4mvpvx50lemh05unarpr5, duration: 11.085µs]
[ 1.768] INFO fabric/router/handler_link.(*bindHandler).BindChannel: {linkId=[5q2jdteIEC7dEMJO07qBu5] routerId=[5xnOn.aTq] routerVersion=[v0.25.13]} link destination support heartbeats
[ 1.768] INFO fabric/router.(*xlinkAccepter).Accept: accepted new link [l/5q2jdteIEC7dEMJO07qBu5]
[ 1.768] INFO fabric/router/handler_ctrl.(*dialHandler).handle |link, linkDialer|: {address=[tls:162354cb-06b0-4953-bc2a-1b10267ad4f4.production.netfoundry.io:6262] linkProtocol=[tls] routerVersion=[v0.25.13] linkId=[5q2jdteIEC7dEMJO07qBu5] routerId=[5xnOn.aTq]} link registered [ 2.587] INFO edge/tunnel/dns.NewDnsServer: dns server running at 10.250.50.78:53
*[ 2.588] INFO edge/tunnel/dns.(resolver).AddHostname: adding ziti-tunnel.resolver.test = 19.65.28.94 to resolver
*[ 2.619] INFO edge/tunnel/dns.(resolver).RemoveHostname: removing ziti-tunnel.resolver.test from resolver [ 4.652] INFO edge/tunnel/intercept.SetDnsInterceptIpRange: dns intercept IP range: 100.64.0.1 - 100.127.255.254
I will try on another host and see if that makes a difference. Also if the lan ip is a public ip would it make a difference? Since in the end state, I would like to use this edge router as a way to get onto ziti network where others servers from RFC1918 address space would be able to use this router without needing Network Address Translation.
If you were going to run a public Ip I would suggest using two interface the private one facing your private Lan clients and the public one facing the internet (This is a similar config to most home router/firewalls. Using a public IP would not change the DNS operation discussed above.
@rcsoleng I tried on a diff host and ran into the same problem. Would it be possible for you to do this using CentOS 8 by any chance? I am wondering if this is some behavior difference of systemd-resolved on Ubuntu vs CentOS/RHEL.
I am neither able to see Current Scopes: DNS on my network interface nor any DNS Servers for it. I am not sure how I can configure DNS resolver at the network interface level.
I’m following along, but I’m not sure, does this mean you got it all figured out now? If not, I was gonna reach out to @rcsoleng and see if he had additional guidance