Getting ziti-edge-tunnel working on RHEL 8

I am trying to install ziti-edge-tunnel on RHEL8. I followed https://support.netfoundry.io/hc/en-us/articles/360045177311-How-to-Use-the-Linux-Tunneler to install.

$ ziti-edge-tunnel version
v0.17.35

$ uname -a
Linux ip-x-x-x-x.us-west-2.compute.internal 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

==> /etc/redhat-release <==
Red Hat Enterprise Linux release 8.5 (Ootpa)

I also see following log lines related to ziti-edge-tunnel -

Failed to set DNS configuration: Unit dbus-org.freedesktop.resolve1.service not found.
ERROR ziti-edge-tunnel:utils.c:30 run_command_va() cmd{resolvectl dns tun0 100.64.0.2} failed: 256/0/Success
Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.
Failed to set domain configuration: Unit dbus-org.freedesktop.resolve1.service not found.

Could you please help? Thanks in advance!

Hi @av-dev. Welcome to OpenZiti! Pleasure to have you in the community!

Right now it’s looking like the ziti-edge-tunnel requires you to run systemd-resolved. Is running systemd-resolved an option?

Thank you for the reply. Yes, I did enable systemd-resolved but it looks like it was tripping over DNSSEC, after I disabled DNSSEC explicitly in /etc/systemd/resolved.conf it seems to be working ( i.e. I am able to do resolvectl query )

Glad you got it working! Do these steps reflect what you needed to do?

systemd-resolved is installed but disabled and NetworkManager is in control by default in RHEL8. You can make systemd-resolved the master resolver with these steps.

I use the same approach on my Ubuntu workstation and just verified this works with a fresh RHEL8 VM I launched in Amazon.

This changes the primary libc nameserver in /etc/resolv.conf to use systemd-resolved built-in “stub” nameserver which merges the configuration of systemd-resolved and NetworkManager into a single domain nameserver.

  1. disable DNSSEC
    sudo sed -E -i 's/#?(DNSSEC).*/\1=no/' /etc/systemd/resolved.conf
    
  2. enable and start the systemd-resolved service
    sudo systemctl enable systemd-resolved && sudo systemd start systemd-resolved
    
  3. configure system to use systemd-resolved stub resolver (target file is auto-generated)
    sudo cp -v /etc/resolv.conf{,.orig} && sudo ln -sfvn /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
    
  4. Install the ziti-edge-tunnel RPM
    sudo dnf install ./ziti-edge-tunnel-0.17.35-1.x86_64.rpm
    
  5. Install an identity enrollment token
    cat | sudo tee /opt/ziti/etc/identity/rhel8aws.jwt
    
  6. Start the tunneler
    sudo systemctl start ziti-edge-tunnel.service
    

As a result of the steps above, I now have these configurations.

[ec2-user@ip-172-31-25-45 ~]$ ll /etc/resolv.conf 
lrwxrwxrwx. 1 root root 37 May 12 20:18 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
[ec2-user@ip-172-31-25-45 ~]$ grep DNSSEC /etc/systemd/resolved.conf 
DNSSEC=no
[ec2-user@ip-172-31-25-45 ~]$ systemctl cat ziti-edge-tunnel.service
# /usr/lib/systemd/system/ziti-edge-tunnel.service
[Unit]
Description=Ziti Edge Tunnel
After=network-online.target

[Service]
Type=simple
EnvironmentFile=/opt/ziti/etc/ziti-edge-tunnel.env
ExecStartPre=/opt/ziti/bin/ziti-edge-tunnel.sh
ExecStart=/opt/ziti/bin/ziti-edge-tunnel run --verbose=${ZITI_VERBOSE} --dns-ip-range=${ZITI_DNS_IP_RANGE} --identity-dir=${ZITI_IDENTITY_DIR}
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
[ec2-user@ip-172-31-25-45 ~]$ resolvectl status tun0
Link 7 (tun0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 100.64.0.2
         DNS Servers: 100.64.0.2
[ec2-user@ip-172-31-25-45 ~]$ resolvectl domain
Global:
Link 7 (tun0):
Link 2 (eth0): us-west-1.compute.internal
[ec2-user@ip-172-31-25-45 ~]$ resolvectl dns
Global:
Link 7 (tun0): 100.64.0.2
Link 2 (eth0): 172.31.0.2

yeah my setup looks almost similar except the ziti-edge-tunnel systemd config. I didnt install ziti-edge-tunnel from RPM but downloaded the zip file from github releases.

1 Like

Thanks for confirming that. The RPM and DEB haven’t been organized under releases yet because the plan is to deliver them in distribution repos. For now it’s possible to look them up in GitHub with a query like this one: Actions · openziti/ziti-tunnel-sdk-c · GitHub

:point_up: this link will be out-of-date when the next release drops, but it illustrates how to compose the query.

1 Like

I removed the manual binary and installed the RPM from your link. I am seeing an error which I believe I didnt see before.

ERROR ziti-sdk:ziti.c:218 ziti_init_opts() /github/workspace/build/_deps/ziti-sdk-c-src/library/ziti.c:188 - load_config(options->config, &cfg) => -13 (Configuration is invalid)

ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:633 load_id_cb() identity[/opt/ziti/etc/identity/client-tunneler.jwt.enrolled] failed to load: failed to initialize ziti

And my endpoint is showing offline in the console.

Although restart seem to have fixed it and my endpoint is online again. I do keep seeing following error messages on the endpoint

ERROR ziti-sdk:channel.c:865 on_channel_connect_internal() ch[0] failed to connect [-125/operation canceled]
INFO ziti-sdk:channel.c:730 reconnect_channel() ch[0] reconnecting in 66927 ms (attempt = 4)

Where is it trying to connect to? I am wondering if my NACL / Security Group rules are blocking something?

TIA.

It’ll be trying to connect to either your controller, or to any edge routers it’s authorized to use.

Sounds like you should do a telnet and figure out if ports (80 &443) are open / shut outbound.

@av-dev You can ignore these non-fatal error messages. I recognize them as having arisen from the fact that there are some files in the tunneler's identity-dir that are not actually identities. As long as the actual identity was loaded then you're good. As an aside, you've reminded me that we should stop saving the enrollment token with a filename suffix like .enrolled after it's used. It's not useful because the fact of the identity JSON file is evidence enough that enrollment succeeded, and we might as well just delete the JWT file.

1 Like

@av-dev Echoing the others and to summarize: it sounds like your endpoint is functioning as expected. Your non-Ziti apps running in RHEL8 should be able to resolve domain names and connect to the remote apps over Ziti normally. You’ll still see non-fatal ERROR level messages in the log whenever internet weather happens, one of the configured routers is momentarily unavailable, etc.

Do you see any more problems that need solving?

Thank you so much for your replies. They have been super helpful.
I am trying to troubleshoot why I cant reach the edge router hosted in my aws account, although I can reach the NetFoundry hosted router.
My Network reachability analyzer is reporting that access is good wrt NACL / Security Groups when ran for source=my rhel 8 server and destination= my edge router instance and port 443.
I could use some tips / suggestions if you have any.

I made a mistake earlier, you need outbound 80, 443 and 6262 (when using the NetFoundry implementation of OpenZiti)… if one is missing that could be why

my outbound rules are very generous to allow all of the above. Do I need to allow additional incoming? on the router host I dont see anything running on 80 ( I didnt select the listener option while launching the router ). Will that cause any issue?

No sir! :slight_smile: OpenZiti is all about closing all inbound firewalls! :slight_smile: The edge router will need one or two ports open by default (443/80). One port is for edge traffic (from like, ziti-edge-tunnel) and one port is for other edge routers to link to this router (if it's configured as a link listener)

1 Like

:slight_smile: 443 is definitely open on the NACL as well as SG but I never see connected message on my endpoints the way I see for NetFoundry hosted router.

Sample logs given below -

INFO tunnel-cbs:ziti_tunnel_ctrl.c:702 on_ziti_event() ztx[aws-server] added edge router router-aws@tls://<my-edge-router-private-ip>:443@<my-edge-router-private-ip>
INFO tunnel-cbs:ziti_tunnel_ctrl.c:702 on_ziti_event() ztx[aws-server] added edge router nf-hosted-router@tls://<nf-edge-router-public-ip>:443@<nf-edge-router-public-ip>
INFO ziti-sdk:channel.c:629 hello_reply_cb() ch[1] connected. EdgeRouter version: v0.25.2|6d2cb18934cc|2022-03-11T16:14:48Z|linux|amd64
INFO tunnel-cbs:ziti_tunnel_ctrl.c:706 on_ziti_event() ztx[aws-server] router nf-hosted-router@tls://<nf-edge-router-public-ip>:443 connected

Can we level-set now and go back to what the root problem is at this point? Before it was ziti-edge-tunnel doesn't work... :slight_smile: you're past that now...

I am trying to troubleshoot why I cant reach the edge router hosted in my aws account

Are you stating that you don't see ziti-edge-tunnel reaching out to the edge-router you deployed? If that's the case, I expect the problem is that there's no edge-router policy in place granting your identity access to that particular router. If that's is the situation, can you check your edge router policy?

yes, that problem of ziti-edge-tunnel not working was resolved after enabling systemd-resolved and disabling DNSSEC.
Now I am trying to make sure my endpoint can talk to my edge router. And we are seeing something interesting.

my edge router firewall is blocking my service ip and if i check firewall rules it only allows 443/tcp for the subnet its deployed in. Whereas my service is deployed in a diff subnet.

Is that a known thing?

As far as edge router policies go, my edge router has association with my service endpoint.

Well that'd be 'a problem' for sure. So without using ziti-edge-tunnel, it sounds to me like you wouldn't even be able to use openssh -s_client -connect command? For example here's an attempt to access one of my edge routers hosted in AWS:

openssl s_client -connect ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8443

That sort of command needs to succeed. (probably stating the obvious there, eh?) :slight_smile:

It sounds to me like your SG is preventing public/external access. Your edge routers really need to allow inbound traffic to your two ports (data/link).

It's not a known AWS issue per-se though. Just depends on how the SG/ACLs are setup I think?