Hi Steven!
If the binary can
dlopen()libsystemd on your system, then the hard-coded path dependencies have no effect, except with regards to inspecting/etc/resolv.conf. After doing a quick test, it seems thatrealpath("/etc/resolv.conf", ...)should canonicalize through multiple levels of symbolic links. Are you seeing something different or perhaps I’m misunderstanding the problem space?
I agree; realpath yields the base file. I think you've covered everything; I'll try and clarify a bit more below, but it seems I'm pretty good to go.
I saw in the Static binary artifacts for CI releases thread, that you attempted to solve this by using
patchelf --add-needed, but I’m wondering if this should’ve been some form ofpatchelf --add-rpath ...? After some googling, maybe the rpath here should be the same as the-Lpath which results from runningnix-shell -p systemd pkg-config --run 'pkg-config --libs libsystemd'?
Initially I was getting these WARN and ERRORs in tunneler log output before patching with an --add-needed:
[ 0.000] INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.19.10)
[ 0.000] INFO tunnel-cbs:ziti_dns.c:172 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
[ 0.022] INFO ziti-edge-tunnel:resolvers.c:429 is_systemd_resolved_primary_resolver() Detected systemd-resolved is primary system resolver
[ 0.022] INFO ziti-edge-tunnel:resolvers.c:65 init_libsystemd() Initializing libsystemd
[ 0.022] WARN ziti-edge-tunnel:resolvers.c:87 init_libsystemd() Failure during dynamic loading function: libsystemd.so.0: cannot open shared object file: No such file or directory
[ 0.022] ERROR ziti-edge-tunnel:tun.c:251 find_dns_updater() could not find a way to configure system resolver. Ziti DNS functionality will be impaired
[ 0.035] ERROR ziti-edge-tunnel:utils.c:31 run_command_va() cmd{grep -q '^nameserver 100.64.0.2' /etc/resolv.conf} failed: 256/2/No such file or directory
Stracing the tunneler app showed it trying to walk a number of paths searching for libsystemd.so.0. Patching with the --add-needed option and re-checking strace showed that it found the lib, and the WARN error disappeared. I just tried with --add-rpath approach instead and strace shows the same positive effect. The binary is a few hundred bytes smaller with the --add-rpath method, but otherwise behavior is the same AFAICT.
What was the error you were seeing after the patches you made in the other post?
After these two patches while using systemd-resolve, I don't see any other unexpected behavior. When not using systemd-resolve with the patches, I receive:
[ 0.000] INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.19.10)
[ 0.000] INFO tunnel-cbs:ziti_dns.c:172 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
[ 0.012] ERROR ziti-edge-tunnel:tun.c:251 find_dns_updater() could not find a way to configure system resolver. Ziti DNS functionality will be impaired
[ 0.017] ERROR ziti-edge-tunnel:utils.c:31 run_command_va() cmd{grep -q '^nameserver 100.64.0.2' /etc/resolv.conf} failed: 256/2/No such file or directory
Despite the grep error above, the tunneler does inject a nameserver 100.64.0.2 entry into /etc/resolv.conf. The grep error then does not show up on subsequent tunneler restarts in this case although the Ziti DNS functionality will be impaired error continues. I'm not looking at the code at the moment, but suspect that is just because the tunneler is looking for and not finding systemd-resolve.
With regards to the fallback to the binaries, as written, the easiest path would be to allow them to be overriden at build time. With the code as written, the idea of searching $PATH would have to be carefully implemented. Also, a major hurdle in our case is that we use CMake’s CPack to package the ZET for the other formats, and I couldn’t find a generator for NixOS format.
Ah, ok. Multiple non-trivial barriers there.
FYI, the error in dlopen() results in the fallback to hard-coded paths, but does not interrupt application startup. In the case that the hardcoded paths cannot be found, it should still fallback to manipulating
/etc/resolv.confdirectly (though this portion is naive in flawed at this time).
Yes, it does appear to do that per above.
Thank you very much for looking into this. From what I see, with systemd-resolve enabled, I don't see issues after the two patches, and without systemd-resolve, while there might be an issue with the grep command indicated by the log error, it doesn't appear to affect functionality so far. We'll get anything else addressed in the nix build once we get there.
Ditto with the other binaries/repos -- we'll aim to address similar issues in the nix build. If something comes up that is otherwise a blocker in that respect, I'll make another thread.
Thanks!
John