I don't have any AV/endpoint protection. Besides, curl -k https://ziti-api.mydomain.com:443/network-jwts works as expected.
I sent you creds by DM.
this error indicates that your IdP is not trusted by system CA (not ziti controller)
Is there a way to verify this using a different path or an external tool? Why is the ziti cli working?
FWIW, I enabled more verbose logging, and certificate validation fails before the OIDC error:
(54974)[ 42.467] TRACE tlsuv:tls_link.c:242 io buffering 7 bytes
(54974)[ 42.467] WARN tlsuv:engine.c:837 80CF9A3FEB7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../src/nssl-3.5.0-3b91ebed43.clean/ssl/statem/statem_clnt.c:2107:
(54974)[ 42.467] ERROR tlsuv:engine.c:868 openssl: handshake was terminated: SSL routines
(54974)[ 42.467] TRACE tlsuv:tls_link.c:222 flushing 7 bytes
(54974)[ 42.467] ERROR tlsuv:tls_link.c:113 TLS(0x5562406c7b68) handshake error certificate verify failed
(54974)[ 42.467] ERROR tlsuv:http.c:192 handshake failed status[3]: certificate verify failed
(54974)[ 42.467] VERBOSE tlsuv:http.c:373 closing connection
(54974)[ 42.467] ERROR ziti-sdk:external_auth.c:33 ext_oath_cfg_cb() ztx[1] OIDC provider configuration failed: (null)
(54974)[ 42.467] ERROR tunnel-cbs:ziti_tunnel_ctrl.c:1098 on_ziti_event() ztx[/opt/openziti/etc/identities/entra.json//opt/openziti/etc/identities/entra.json] authorization flow cannot continue: (null)
(54974)[ 42.467] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:460 on_event() ztx[/opt/openziti/etc/identities/entra.json] context event : status is (null)
(54974)[ 42.467] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:514 on_event() ztx[/opt/openziti/etc/identities/entra.json] failed to connect to controller due to (null)
(54974)[ 42.467] TRACE tlsuv:tls_link.c:185 closing TLS link
My system is on openssl 3.5.0.
Probably the best way to test instead of curl would be to use openssl s_client -connect.
So something like this:
openssl s_client -connect fqdn.here:port
Can you -connect to your IdP as well? and does that succeed?
For Entra this is Microsoft and I have no issues connecting with openssl s_client -connect login.microsoftonline.com:443.
Connecting to 40.126.53.17
CONNECTED(00000003)
depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
verify return:1
depth=1 C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA
verify return:1
depth=0 C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=stamp2.login.microsoftonline.com
verify return:1
...
what hardware and linux distro are you using?
How do you install ZET?
can you try openssl s_client.. command as root (with sudo)?
Edit: more questions
I'm using NixOS on a workstation with 64GB RAM and an AMD Ryzen 9 processor. I created a nix package to install the binaries downloaded from releases:
openziti-tools = prev.stdenv.mkDerivation rec {
pname = "openziti-tools";
version = ziti-rev;
srcBinZiti = builtins.fetchTarball {
url = "https://github.com/openziti/ziti/releases/download/v${ziti-rev}/ziti-linux-amd64-${ziti-rev}.tar.gz";
sha256 = "sha256:0s0lbh7cd0c4hmdiaa6cmlddri2ggx020wv3d2b34yazz2jflzfn";
};
srcBinZitiEdgeTunnel = prev.fetchzip rec {
version = ziti-edge-tunnel-rev;
url = "https://github.com/openziti/ziti-tunnel-sdk-c/releases/download/v${ziti-edge-tunnel-rev}/ziti-edge-tunnel-Linux_x86_64.zip";
hash = "sha256-dCikxZuliBhTgPEeDcYVdOhnlz7P5InhmGpxksBX0hk=";
};
src = srcBinZiti;
nativeBuildInputs = with prev; [
autoPatchelfHook
makeWrapper
unzip
zlib
];
runtimeDeps = with prev; [
systemd
];
installPhase = ''
install -m755 -d $out/bin/
install -m755 -D ./ziti $out/bin/
install -m755 -D $srcBinZitiEdgeTunnel/ziti-edge-tunnel $out/bin/.ziti-edge-tunnel-unwrapped
makeWrapper $out/bin/.ziti-edge-tunnel-unwrapped $out/bin/ziti-edge-tunnel \
--set LD_LIBRARY_PATH ${prev.lib.makeLibraryPath runtimeDeps}
'';
};
I'll try and build them from source too when I have more time.
Here's the openssl output with sudo:
> sudo openssl s_client -connect login.microsoftonline.com:443 2>&1 </dev/null | grep Verify
Verify return code: 0 (ok)
ok this interesting. I am not familiar with nix packages, but it is possible that ZET package is isolated from the system CA bundle files?
Does it need access to system CA bundle here?
The CA bundle is globally accessible, but I can investigate it.
Is the CA bundle relevant to this issue? The controller CA is self-signed and won't be in any system's trust store unless specifically added. The identity has the controller root CA in its config, so I assume the tunneler should use that to validate the certificate, not the OS's CA bundle.
the error that you see it from ziti-sdk querying your IdP(entra). it does not use private ziti bundle for that, and relies on the system CA.
can run ZET under strace? you should see something like this
openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = 13
1 Like
I think you found the issue!
root@kipchoge:~/ > grep ssl ziti_trace.log | grep pem
1406538 openat(AT_FDCWD, "/etc/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)
root@kipchoge:~/ > ls /etc/ssl/certs
ca-bundle.crt ca-certificates.crt
The trust bundle is in ca-bundle.crt. I guess I can fix it with a symlink. Alternatively, what determines what path is used?
Awesome! The symlink fixes the issue!
1 Like
I believe the path comes from openssl configuration or it may be hardcoded inside openssl library
can you mark this topic as Solved please?
Here's what Gemini replies with:
The path /etc/ssl/cert.pem is a compile-time default that is hardcoded into the SSL
library (like OpenSSL) when it is built. The strace output you're seeing is the
ziti-edge-tunnel process asking the operating system for the file at the location its
SSL library was told to look for.Here’s why it's looking there and not at the NixOS path:
- Pre-compiled Binaries: In your flake.nix, you are downloading a pre-compiled version of
ziti-edge-tunnel directly from GitHub (fetchTarball, fetchzip). This binary was
compiled and linked on a standard Linux distribution (like Debian, Ubuntu, or Alpine),
where a common default path for the CA bundle is /etc/ssl/cert.pem.- NixOS Specifics: The NixOS OpenSSL package is specifically patched to look for the CA
bundle at /etc/ssl/certs/ca-bundle.crt. When you compile software from source on NixOS,
it links against this Nix-aware version of OpenSSL and automatically knows the correct
path.- The Conflict: Because you are running a pre-compiled binary, it doesn't know it's on
NixOS. It only knows the default path that was baked into it when it was originally
compiled, leading to the ENOENT (No such file or directory) error you observed.
So I think building this from source might solve this in the best way for NixOS.
Thank you so much for all your help @ekoby and @TheLumberjack!
1 Like
That's good to know for the future. If you do build it from source and that also solves the issue let us know. Until then, future readers can make that symlink and be up and running in short order. 
Thanks for following up with that. Cheers
1 Like
I was jumping up and down earlier today because I managed to package ZTE and build it successfully from source: nixdots/modules/nixos/opt/openziti/default.nix at main · rochecompaan/nixdots · GitHub. Additionally, I had no CA and cert troubles either! Consider this a WIP since I'm still testing if everything works end-to-end.
I encountered the same issue mentioned in the hard-coded paths discussion here: Hard coded executable paths. I fixed this by patching paths to use Nix package paths.
The other challenge was to pull in all the dependencies using Nix fetchers instead of vcpkg, but I compiled the list of dependencies by reviewing the various CMakeLists.txt files.
I'll start a new topic about the Nix package once I'm done.