Permission issue

carljmosca · July 2, 2023, 4:49pm

Installed ziti-edge-tunnel via dnf on Rocky9. Seeing a permission denied issue writing a temporary file to /etc when the Ziti DNS updates /etc/resolv.conf

Jul 02 11:45:49 openziti.server systemd[1]: Starting Ziti Edge Tunnel...
Jul 02 11:45:49 openziti.server ziti-edge-tunnel.sh[1674]: NOTICE: no new JWT files in /opt/openziti/etc/identities/*.jwt
Jul 02 11:45:49 openziti.server systemd[1]: Started Ziti Edge Tunnel.
Jul 02 11:45:49 openziti.server ziti-edge-tunnel[1675]: (1675)[        0.039]    WARN ziti-edge-tunnel:resolvers.c:351 try_libsystemd_resolver() libsystemd resolver unsuccessful. Falling back to legacy resolvers
Jul 02 11:45:49 openziti.server ziti-edge-tunnel[1675]: (1675)[        0.047]    WARN ziti-edge-tunnel:tun.c:255 find_dns_updater() Adding ziti resolver to /etc/resolv.conf. Ziti DNS functionality may be impaired
Jul 02 11:45:49 openziti.server ziti-edge-tunnel[1695]: sed: couldn't open temporary file /etc/sedzuhw6Q: Permission denied
Jul 02 11:45:49 openziti.server ziti-edge-tunnel[1675]: (1675)[        0.052]   ERROR ziti-edge-tunnel:utils.c:31 run_command_va() cmd{sed -z -i 's/nameserver/nameserver 100.64.0.2\nnameserver/' /etc/resolv.conf} failed: 1024/2/No such file or directory
Jul 02 12:14:50 openziti.server ziti-edge-tunnel[1675]: (1675)[     1740.241]    WARN ziti-sdk:bind.c:312 on_message() binding failed: -17/ziti edge router is not available
Jul 02 12:14:50 openziti.server ziti-edge-tunnel[1675]: (1675)[     1740.241]    WARN ziti-sdk:bind.c:312 on_message() binding failed: -17/ziti edge router is not available

carljmosca · July 2, 2023, 7:39pm

I am seeing the same issue on a new install of Rocky 8.

TheLumberjack · July 3, 2023, 12:40pm

This might be related to the recent v0.21 release Release v0.21.0 · openziti/ziti-tunnel-sdk-c · GitHub

Are you using an installer or are you running it from a direct download? The release notes (linked above) have some helpful comments that might set you straight. If not, let us know.

carljmosca · July 3, 2023, 4:33pm

I am installing via RPM as documented here. I have also downloaded a couple of immediately prior releases of the RPM manually and noticed the same issue.

With respect to the Release v0.21.9 provided link, was the intent to write the temporary file for the editing of the /etc/resolv.conf file? It seems to me that the error is a result of a lack of permission for the /etc directory files while that change refers to the /tmp/.ziti directory.

I did a fresh install on an Ubuntu VM this morning and I am not seeing the same error but it does not appear that the /etc/resolv.conf file is modified.

qrkourier · July 5, 2023, 4:05pm

Hey there @carljmosca, Those are the best instructions for installing the RPM

The ziti-edge-tunnel process attempts to configure a Ziti DNS resolver when it starts. On systems that installed the tunneler with the RPM or DEB package (runs as user “ziti”) and have systemd-resolved, it will succeed at auto-configuring Ziti DNS.

Auto-configuring Ziti DNS didn’t work in your case because Rocky/RedHat most likely is using NetworkManager, not systemd-resolved. For RedHat-flavored Linux it’s currently necessary to work around the problem with one of these solutions:

run-as user “root” instead of user “ziti” by changing the service unit in /opt/openziti/share/ziti-edge-tunnel.service and running sudo systemctl daemon-reload. This grants permission to munge /etc/resolv.conf.
Or, separately configure a DNS resolver for the Ziti nameserver address (default is 100.64.0.2), optionally constrained by routing/search domain namespace.

Assuming your RedHat/Rocky system is using NetworkManager, you may wish to modify the NM connection with nmcli.
```
# you must substitute the correct connection name for
#  "Wired Connection 1", and 1.1.1.1 is an example of 
#  a recursive nameserver for answering non-Ziti queries
(
set -euxo pipefail;
CONN="Wired Connection 1";
nmcli connection modify "$CONN" \
  ipv4.ignore-auto-dns yes \
  ipv4.dns "100.64.0.2,1.1.1.1";
nmcli connection up "$CONN"
)
```
Note: The run --dns-upstream=1.1.1.1 option allows you to specify a recursive nameserver so that all queries may be directed to the Ziti nameserver (ref).

I have raised a GitHub issue to track this in RedHat Linux fails to auto-configure Ziti DNS · Issue #685 · openziti/ziti-tunnel-sdk-c · GitHub.

carljmosca · July 6, 2023, 10:30am

Thank you @qrkourier - I will give one of (or both) the provided solutions a try. I was actually thinking this is not a currently a show-stopper but I noticed it when trying to resolve (pun intended) another issue I am seeing on multiple flavors of Linux (Rocky 8 and 9 and Ubuntu). Using one of the simple examples for http traffic, everything seems to work nicely and then a few hours later, it no longer works. I will try to sort out the DNS issue first and then open a separate issue if this persists.

carljmosca · July 6, 2023, 3:24pm

As expected, changing the service to run as root eliminates the issue writing to /etc (tmp file in /etc and editing /etc/resolv.conf) - but I am still seeing the other (I believe unrelated) issue. I will open a new issue. - Thank you @qrkourier

qrkourier · July 6, 2023, 3:43pm

You’re welcome!

Your problem with Ziti services that stop working after some time is interesting. I suspect the Ziti DNS resolver configuration isn’t sticky and is somehow clobbered by another process managing DNS on the same host.

This shouldn’t happen on systems that use systemd-resolved. You can see the Ziti DNS resolver configuration on those systems like this:

$ resolvectl dns tun0                                 
Link 98 (tun0): 100.64.0.2

$ resolvectl domain tun0
Link 98 (tun0): ~.

This output indicates that systemd-resolved should route DNS queries for all domains (~. wildcard-top-level) to the nameservers provided by the link bound to the tun0 interface.

If the tunneler process is making a one-time change to /etc/resolv.conf at startup then it’s possible another process too is managing the nameservers in that file and clobbering the tunneler’s nameserver directive.

carljmosca · July 6, 2023, 3:57pm

That all makes sense but I am seeing the same failure on an Ubuntu machine for which no changes are made/necessary to /etc/resolv.conf - perhaps I should spend more time on the Ubuntu VM to track this one down.

qrkourier · July 6, 2023, 4:05pm

True, there may be multiple conspiring issues at work. It will be telling if the Ziti services that stop working after some time fail because the client app cannot resolve the Ziti service address in DNS.

It would be amazing if you can reproduce the issue in Ubuntu. I use an Ubuntu-based OS for my workstation and haven’t tripped over that problem yet.

carljmosca · July 6, 2023, 5:06pm

Guessing I will need to increase the verbosity of the log settings but this is what I am seeing on the Ubuntu VM. (This particular log excerpt is while it is still working and the fqdn/ip has been redacted.) I have not been able to discern differences after the failure but I have not yet made time to increase verbosity and then look. So far it seems like once it fails, there is no (automatic) recovery but it’s quite possible I have not waited long enough.

Jul 06 16:53:40 util02 ziti-edge-tunnel[2270072]: (2270072)[      116.594]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:40 util02 ziti-edge-tunnel[2270072]: (2270072)[      116.594]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)
Jul 06 16:53:35 util02 ziti-edge-tunnel[2270072]: (2270072)[      111.536]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:35 util02 ziti-edge-tunnel[2270072]: (2270072)[      111.536]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)
Jul 06 16:53:30 util02 ziti-edge-tunnel[2270072]: (2270072)[      106.482]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:30 util02 ziti-edge-tunnel[2270072]: (2270072)[      106.482]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)
Jul 06 16:53:25 util02 ziti-edge-tunnel[2270072]: (2270072)[      101.410]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:25 util02 ziti-edge-tunnel[2270072]: (2270072)[      101.410]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)
Jul 06 16:53:20 util02 ziti-edge-tunnel[2270072]: (2270072)[       96.347]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:20 util02 ziti-edge-tunnel[2270072]: (2270072)[       96.347]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)
Jul 06 16:53:15 util02 ziti-edge-tunnel[2270072]: (2270072)[       91.279]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:15 util02 ziti-edge-tunnel[2270072]: (2270072)[       91.279]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)
Jul 06 16:53:10 util02 ziti-edge-tunnel[2270072]: (2270072)[       86.221]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:10 util02 ziti-edge-tunnel[2270072]: (2270072)[       86.221]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)
Jul 06 16:53:05 util02 ziti-edge-tunnel[2270072]: (2270072)[       81.152]    WARN ziti-sdk:ziti.c:1437 api_session_cb() ztx[0] failed to get api session from ctrl[https://some-ip-or-fqdn:8441] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
Jul 06 16:53:05 util02 ziti-edge-tunnel[2270072]: (2270072)[       81.152]   ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[some-ip-or-fqdn] request failed: -103(software caused connection abort)

carljmosca · July 6, 2023, 5:10pm

I realize I really should move this to a new issue because the permission is seemingly unrelated.

Topic		Replies	Views
Questions about Linux tunnelers	2	269	May 20, 2023
Binding failed ziti edge router not available Ziti Edge Tunnel	7	184	July 15, 2023
DNS resolver not working with Docker Ziti Edge Tunnel	7	323	June 5, 2023
Ziti identies are failing after upgrade of edge tunneler	23	116	October 22, 2024
SEGV crashing ziti-edge-tunnel von v0.22.15	1	98	February 15, 2024

Permission issue

Related topics