Hard coded executable paths

johnalotoski · October 1, 2022, 12:11am

Hello again!

I haven’t poked through the source code much yet, but I did see some hard coded paths for binaries included in header definitions. Example here where we have:

#define BUSCTL "/usr/bin/busctl"
#define RESOLVCONF "/usr/sbin/resolvconf"
#define RESOLVECTL "/usr/bin/resolvectl"
#define SYSTEMD_RESOLVE "/usr/bin/systemd-resolve"

Even though NixOS may have all of these binaries present and in the system environment path (I do in this case), none of the code utilizing these header definitions will work because of the path differences from FHS:

❯ for i in busctl resolvconf resolvectl systemd-resolve; do
  echo "Finding $i in path and resolved physical location"
  which $i
  readlink -e $(which $i)
  echo
done

Finding busctl path and physical location
/run/current-system/sw/bin/busctl
/nix/store/lwkp9z6w20yvavz6g87gfiglbp5xagb0-systemd-250.4/bin/busctl

Finding resolvconf path and physical location
/run/current-system/sw/bin/resolvconf
/nix/store/lwkp9z6w20yvavz6g87gfiglbp5xagb0-systemd-250.4/bin/resolvectl

Finding resolvectl path and physical location
/run/current-system/sw/bin/resolvectl
/nix/store/lwkp9z6w20yvavz6g87gfiglbp5xagb0-systemd-250.4/bin/resolvectl

Finding systemd-resolve path and physical location
/run/current-system/sw/bin/systemd-resolve
/nix/store/lwkp9z6w20yvavz6g87gfiglbp5xagb0-systemd-250.4/bin/resolvectl

Perhaps required binaries could be found via $PATH search first, and then the use of hard-coded FHS paths as a fallback if needed?

I think the two functions is_systemd_resolved_primary_resolver and is_resolvconf_systemd_resolved don’t work for similar reasons, although in this case they are just static files, not binaries in the search path.

Just taking a quick look at the resolved paths on NixOS 22.05, we have two symlinks of indirection to the ziti tunneler expected hard-coded resolve path, so the function comparisons fail:

# With resolved enabled
❯ ls -la /etc/resolv.conf && echo && systemd-resolve --status | head
lrwxrwxrwx 1 root root 23 Sep 30 19:00 /etc/resolv.conf -> /etc/static/resolv.conf

Global
           Protocols: +LLMNR +mDNS -DNSOverTLS
                      DNSSEC=allow-downgrade/unsupported
    resolv.conf mode: stub
  Current DNS Server: 8.8.8.8
         DNS Servers: 8.8.8.8 192.168.1.1
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google
                      1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google
                      2606:4700:4700::1111#cloudflare-dns.com
                      2001:4860:4860::8888#dns.google

# ^^^ In this case it's 2 links of indirection to the expected hard-coded path:
❯ ls -la /etc/static/resolv.conf 
lrwxrwxrwx 1 root root 37 Dec 31  1969 /etc/static/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

# ------------------

# Without resolved enabled
❯ ls -la /etc/resolv.conf && systemd-resolve --status | head
-rw-r--r-- 1 root root 117 Sep 30 19:04 /etc/resolv.conf

Failed to get global data: Unit dbus-org.freedesktop.resolve1.service not found.

sabedevops · October 1, 2022, 12:40am

Hey @johnalotoski, thanks for the feedback. There are a few solutions we could implement here, and I’d like to understand a few fine points before attempting one since I’m not as familiar with NixOS.

Are the links /run/current-system/sw/bin/[binary] always installed to some /nix/store/.../bin/[binary]? Similarly, can you describe the purpose of /etc/static in some greater detail for NixOS?

I’m trying to understand whether the solution here is to change these hardcoded paths to depends on some #ifndef pattern, as opposed to searching $PATH. We could potentially then handle this in system packaging, or allow the user to define their own during build time.

What do you think?

johnalotoski · October 1, 2022, 9:38pm

Hi @sabedevops! Sure, happy to answer questions on NixOS; it is rather different from most other distros.

Are the links /run/current-system/sw/bin/[binary] always installed to some /nix/store/.../bin/[binary] ?

The nix store typically begins with a prefix of /nix/store/, and while it is possible to change it, that tends to be fairly rare except under special use cases and is generally not advised as doing so prevents the use of community binary caches.
The general pattern for binaries is for to be placed in /nix/store/$HASH/bin/$BINARY. However, there are exceptions to this. Checking my own /run/current-system/sw/bin/ directory, for example, I have:
- 1828 binaries from 320 different installed packages
- 1795 of these binaries are found in the .../bin/ subdir
- 33 binaries are found in other subdirectories (about 1.8%)
- The exceptions in my case tend to be special build executables (/share/... subdir) or library tooling related executables (/libexec/... subdir)

Similarly, can you describe the purpose of /etc/static in some greater detail for NixOS?

Linking declarative /etc files in a static subdir helps implement an atomic upgrade and rollback feature in NixOS.

I’m trying to understand whether the solution here is to change these hardcoded paths to depends on some #ifndef pattern, as opposed to searching $PATH. We could potentially then handle this in system packaging, or allow the user to define their own during build time. What do you think?

Some of what I’m thinking would depend on whether OpenZiti would like to have the release artifacts work directly on NixOS. Probably the answer to that may depend on level of effort, right? Maybe yes if level of effort is low, maybe not so much if level of effort is high? There are probably 3 levels of NixOS compatibility to consider with regard to direct use of the release binaries:

Binaries work on NixOS as downloaded from the repo:
- libs are compatible (statically linked), code is compatible (no FHS code deps and logic)
Binaries don’t work on NixOS (reason: lib refs only):
- libs are incompatible due to FHS paths, but can be patched easily directly from release binary with minimal effort
- code is compatible (no FHS code deps and logic)
Binaries don’t work on NixOS (reason: FHS code deps and logic [and optionally lib ref problem also]):
- code is incompatible (due to FHS code deps and logic)
- optionally: libs are incompatible due to FHS paths, but can be patched easily directly from release binary with minimal effort

Right now, we are in case (3) for the tunneler client. I haven’t look at the other binaries yet, but probably they are case (3) also. If the code logic adjustments are addressed at the runtime level, then we are in case (2) and it would be fairly trivial for a NixOS user to download a release binary, patch the lib refs and directly use the binary. Case (1) would probably only be achieved if the builds are static like mentioned in the other thread here as well as the FHS code logic issue being handled at runtime.

If the hard coded paths are handled by #ifndef at build time, I think that pretty much guarantees NixOS incompatibility with release binaries because presumably the CI binary artifact releases will continue to be built with FHS paths, right?

Ultimately, we’d like to see OpenZiti packaged for Nix, in which case the tunneler code will work with or without #ifndef addition, because the libs will be appropriately linked and the FHS paths appropriately substituted for NixOS compatible paths during the build. Downside will be that NixOS users won’t be able to use the upstream release binaries directly if they want, although I think that’s probably not unusual on NixOS for these very reasons. It just may be awhile before we (or someone else) gets the time to package OpenZiti for Nix.

John

sabedevops · October 4, 2022, 11:42pm

Hey John,

I finally had a chance to review the source code that handles this bit, my apologies for the delay.

After looking over the logical flow, I would’ve expected a release package to work using the libsystemd integration in the ziti-edge-tunnel. While libsystemd is needed as a build time dependency of this integration, it is the runtime dependency we are concerned with in this case.

If the binary can dlopen() libsystemd on your system, then the hard-coded path dependencies have no effect, except with regards to inspecting /etc/resolv.conf. After doing a quick test, it seems that realpath("/etc/resolv.conf", ...) should canonicalize through multiple levels of symbolic links. Are you seeing something different or perhaps I’m misunderstanding the problem space?

I saw in the https://openziti.discourse.group/t/static-binary-artifacts-for-ci-releases/790 thread, that you attempted to solve this by using patchelf --add-needed, but I’m wondering if this should’ve been some form of patchelf --add-rpath ...? After some googling, maybe the rpath here should be the same as the -L path which results from running nix-shell -p systemd pkg-config --run 'pkg-config --libs libsystemd'?

What was the error you were seeing after the patches you made in the other post?

With regards to the fallback to the binaries, as written, the easiest path would be to allow them to be overriden at build time. With the code as written, the idea of searching $PATH would have to be carefully implemented. Also, a major hurdle in our case is that we use CMake’s CPack to package the ZET for the other formats, and I couldn’t find a generator for NixOS format.

Thanks,
- Steven

FYI, the error in dlopen() results in the fallback to hard-coded paths, but does not interrupt application startup. In the case that the hardcoded paths cannot be found, it should still fallback to manipulating /etc/resolv.conf directly (though this portion is naive in flawed at this time).

johnalotoski · October 5, 2022, 1:37am

Hi Steven!

If the binary can dlopen() libsystemd on your system, then the hard-coded path dependencies have no effect, except with regards to inspecting /etc/resolv.conf . After doing a quick test, it seems that realpath("/etc/resolv.conf", ...) should canonicalize through multiple levels of symbolic links. Are you seeing something different or perhaps I’m misunderstanding the problem space?

I agree; realpath yields the base file. I think you’ve covered everything; I’ll try and clarify a bit more below, but it seems I’m pretty good to go.

I saw in the Static binary artifacts for CI releases thread, that you attempted to solve this by using patchelf --add-needed , but I’m wondering if this should’ve been some form of patchelf --add-rpath ... ? After some googling, maybe the rpath here should be the same as the -L path which results from running nix-shell -p systemd pkg-config --run 'pkg-config --libs libsystemd' ?

Initially I was getting these WARN and ERRORs in tunneler log output before patching with an --add-needed:

[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.19.10)
[        0.000]    INFO tunnel-cbs:ziti_dns.c:172 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
[        0.022]    INFO ziti-edge-tunnel:resolvers.c:429 is_systemd_resolved_primary_resolver() Detected systemd-resolved is primary system resolver
[        0.022]    INFO ziti-edge-tunnel:resolvers.c:65 init_libsystemd() Initializing libsystemd
[        0.022]    WARN ziti-edge-tunnel:resolvers.c:87 init_libsystemd() Failure during dynamic loading function: libsystemd.so.0: cannot open shared object file: No such file or directory
[        0.022]   ERROR ziti-edge-tunnel:tun.c:251 find_dns_updater() could not find a way to configure system resolver. Ziti DNS functionality will be impaired
[        0.035]   ERROR ziti-edge-tunnel:utils.c:31 run_command_va() cmd{grep -q '^nameserver 100.64.0.2' /etc/resolv.conf} failed: 256/2/No such file or directory

Stracing the tunneler app showed it trying to walk a number of paths searching for libsystemd.so.0. Patching with the --add-needed option and re-checking strace showed that it found the lib, and the WARN error disappeared. I just tried with --add-rpath approach instead and strace shows the same positive effect. The binary is a few hundred bytes smaller with the --add-rpath method, but otherwise behavior is the same AFAICT.

What was the error you were seeing after the patches you made in the other post?

After these two patches while using systemd-resolve, I don’t see any other unexpected behavior. When not using systemd-resolve with the patches, I receive:

[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v0.19.10)
[        0.000]    INFO tunnel-cbs:ziti_dns.c:172 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
[        0.012]   ERROR ziti-edge-tunnel:tun.c:251 find_dns_updater() could not find a way to configure system resolver. Ziti DNS functionality will be impaired
[        0.017]   ERROR ziti-edge-tunnel:utils.c:31 run_command_va() cmd{grep -q '^nameserver 100.64.0.2' /etc/resolv.conf} failed: 256/2/No such file or directory

Despite the grep error above, the tunneler does inject a nameserver 100.64.0.2 entry into /etc/resolv.conf. The grep error then does not show up on subsequent tunneler restarts in this case although the Ziti DNS functionality will be impaired error continues. I’m not looking at the code at the moment, but suspect that is just because the tunneler is looking for and not finding systemd-resolve.

With regards to the fallback to the binaries, as written, the easiest path would be to allow them to be overriden at build time. With the code as written, the idea of searching $PATH would have to be carefully implemented. Also, a major hurdle in our case is that we use CMake’s CPack to package the ZET for the other formats, and I couldn’t find a generator for NixOS format.

Ah, ok. Multiple non-trivial barriers there.

FYI, the error in dlopen() results in the fallback to hard-coded paths, but does not interrupt application startup. In the case that the hardcoded paths cannot be found, it should still fallback to manipulating /etc/resolv.conf directly (though this portion is naive in flawed at this time).

Yes, it does appear to do that per above.

Thank you very much for looking into this. From what I see, with systemd-resolve enabled, I don’t see issues after the two patches, and without systemd-resolve, while there might be an issue with the grep command indicated by the log error, it doesn’t appear to affect functionality so far. We’ll get anything else addressed in the nix build once we get there.

Ditto with the other binaries/repos – we’ll aim to address similar issues in the nix build. If something comes up that is otherwise a blocker in that respect, I’ll make another thread.

Thanks!
John

sabedevops · October 5, 2022, 1:46pm

Hey John,

I’ve opened PR 506 to address some of the finer points you’ve surfaced here. The grep command will no longer produce this error, and I’ve downgraded the system resolver configuration message from an ERROR to a WARN. Additionally, it allows the user to override the paths to the binaries at build time, to open the door for this and other distributions which might relocate these packages.

To be clear, we currently support 4 ways of manipulating the system resolver:

libsystemd (systemd-resolved)
resolvectl/systemd-resolve binaries (systemd-resolved)
resolvconf (when NOT a personality of the above binaries)
/etc/resolv.conf direct manipulation

For the default NixOS systems, I believe the best support path is #1 above, with patchelf because it can be built from the release binaries. With this PR, someone building the binary may eventually be able to support #2 and #3 above. Failing both (as it seems by default in the NixOS docker image, which doesn’t ship with systemd or dbus), then #4 above still supplies ground cover.

My intuition here tells me the rpath modification in the elf header is the right approach here, but I’m glad you’ve managed to get it running on NixOS in either case! The libsystemd work wasn’t designed for this use case, but it’s always cool to see when something you’ve done enables a new use case as a happy byproduct like this!

Thanks for your feedback!
- Steven

johnalotoski · October 5, 2022, 2:33pm

Awesome, thank you very much Steven! Also that clarification on resolver manipulation preference is helpful too! I’ll stick with the rpath patch mod preferentially going forward – I’m just about positive you know more about ELF headers than I do .

Ah, if you tried docker pull nixos/nix, that’s a minimal non-NixOS base build with only the Nix package manager which is why systemd and dbus are missing. Vbox install or demo appliance image is probably the next closest thing if you ever want to give it a try.

Thanks!
John