$ ziti-edge-tunnel version
v0.17.35
$ uname -a
Linux ip-x-x-x-x.us-west-2.compute.internal 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
==> /etc/redhat-release <==
Red Hat Enterprise Linux release 8.5 (Ootpa)
I also see following log lines related to ziti-edge-tunnel -
Failed to set DNS configuration: Unit dbus-org.freedesktop.resolve1.service not found.
ERROR ziti-edge-tunnel:utils.c:30 run_command_va() cmd{resolvectl dns tun0 100.64.0.2} failed: 256/0/Success
Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.
Failed to set domain configuration: Unit dbus-org.freedesktop.resolve1.service not found.
Thank you for the reply. Yes, I did enable systemd-resolved but it looks like it was tripping over DNSSEC, after I disabled DNSSEC explicitly in /etc/systemd/resolved.conf it seems to be working ( i.e. I am able to do resolvectl query )
Glad you got it working! Do these steps reflect what you needed to do?
systemd-resolved is installed but disabled and NetworkManager is in control by default in RHEL8. You can make systemd-resolved the master resolver with these steps.
I use the same approach on my Ubuntu workstation and just verified this works with a fresh RHEL8 VM I launched in Amazon.
This changes the primary libc nameserver in /etc/resolv.conf to use systemd-resolved built-in “stub” nameserver which merges the configuration of systemd-resolved and NetworkManager into a single domain nameserver.
disable DNSSEC
sudo sed -E -i 's/#?(DNSSEC).*/\1=no/' /etc/systemd/resolved.conf
[ec2-user@ip-172-31-25-45 ~]$ resolvectl status tun0
Link 7 (tun0)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 100.64.0.2
DNS Servers: 100.64.0.2
[ec2-user@ip-172-31-25-45 ~]$ resolvectl domain
Global:
Link 7 (tun0):
Link 2 (eth0): us-west-1.compute.internal
[ec2-user@ip-172-31-25-45 ~]$ resolvectl dns
Global:
Link 7 (tun0): 100.64.0.2
Link 2 (eth0): 172.31.0.2
yeah my setup looks almost similar except the ziti-edge-tunnel systemd config. I didnt install ziti-edge-tunnel from RPM but downloaded the zip file from github releases.
Thanks for confirming that. The RPM and DEB haven’t been organized under releases yet because the plan is to deliver them in distribution repos. For now it’s possible to look them up in GitHub with a query like this one: Actions · openziti/ziti-tunnel-sdk-c · GitHub
this link will be out-of-date when the next release drops, but it illustrates how to compose the query.
Although restart seem to have fixed it and my endpoint is online again. I do keep seeing following error messages on the endpoint
ERROR ziti-sdk:channel.c:865 on_channel_connect_internal() ch[0] failed to connect [-125/operation canceled]
INFO ziti-sdk:channel.c:730 reconnect_channel() ch[0] reconnecting in 66927 ms (attempt = 4)
Where is it trying to connect to? I am wondering if my NACL / Security Group rules are blocking something?
@av-dev You can ignore these non-fatal error messages. I recognize them as having arisen from the fact that there are some files in the tunneler's identity-dir that are not actually identities. As long as the actual identity was loaded then you're good. As an aside, you've reminded me that we should stop saving the enrollment token with a filename suffix like .enrolled after it's used. It's not useful because the fact of the identity JSON file is evidence enough that enrollment succeeded, and we might as well just delete the JWT file.
@av-dev Echoing the others and to summarize: it sounds like your endpoint is functioning as expected. Your non-Ziti apps running in RHEL8 should be able to resolve domain names and connect to the remote apps over Ziti normally. You’ll still see non-fatal ERROR level messages in the log whenever internet weather happens, one of the configured routers is momentarily unavailable, etc.
Thank you so much for your replies. They have been super helpful.
I am trying to troubleshoot why I cant reach the edge router hosted in my aws account, although I can reach the NetFoundry hosted router.
My Network reachability analyzer is reporting that access is good wrt NACL / Security Groups when ran for source=my rhel 8 server and destination= my edge router instance and port 443.
I could use some tips / suggestions if you have any.
I made a mistake earlier, you need outbound 80, 443 and 6262 (when using the NetFoundry implementation of OpenZiti)… if one is missing that could be why
my outbound rules are very generous to allow all of the above. Do I need to allow additional incoming? on the router host I dont see anything running on 80 ( I didnt select the listener option while launching the router ). Will that cause any issue?
No sir! OpenZiti is all about closing all inbound firewalls! The edge router will need one or two ports open by default (443/80). One port is for edge traffic (from like, ziti-edge-tunnel) and one port is for other edge routers to link to this router (if it's configured as a link listener)
Can we level-set now and go back to what the root problem is at this point? Before it was ziti-edge-tunnel doesn't work... you're past that now...
I am trying to troubleshoot why I cant reach the edge router hosted in my aws account
Are you stating that you don't see ziti-edge-tunnel reaching out to the edge-router you deployed? If that's the case, I expect the problem is that there's no edge-router policy in place granting your identity access to that particular router. If that's is the situation, can you check your edge router policy?
yes, that problem of ziti-edge-tunnel not working was resolved after enabling systemd-resolved and disabling DNSSEC.
Now I am trying to make sure my endpoint can talk to my edge router. And we are seeing something interesting.
my edge router firewall is blocking my service ip and if i check firewall rules it only allows 443/tcp for the subnet its deployed in. Whereas my service is deployed in a diff subnet.
Is that a known thing?
As far as edge router policies go, my edge router has association with my service endpoint.
Well that'd be 'a problem' for sure. So without using ziti-edge-tunnel, it sounds to me like you wouldn't even be able to use openssh -s_client -connect command? For example here's an attempt to access one of my edge routers hosted in AWS:
That sort of command needs to succeed. (probably stating the obvious there, eh?)
It sounds to me like your SG is preventing public/external access. Your edge routers really need to allow inbound traffic to your two ports (data/link).
It's not a known AWS issue per-se though. Just depends on how the SG/ACLs are setup I think?
this link will be out-of-date when the next release drops, but it illustrates how to compose the query.
OpenZiti is all about closing all inbound firewalls!