Getting Authentik and OpenZiti to work with External JWTs

Hello,

As requested, starting a more targeted thread relating to the issues with External JWTs and OIDC Authentication with Authentik.

Environment: Running latest Authentik IDP (like keycloak, authelia, etc.). Authentik is working great for apps, but I'd like to use it authenticate users on my Open Ziti network.
I have upgraded the controller/router/ZAC following the documentation and am on:

Controller: v1.2.2
ZAC: 3.7.1

I had used one of the quick start tutorials on Linux to set this up on an LVM running Debian on my Proxmox host. It's just a simple one router, one controller configuration with ZAC currently. I have an Android and Windows 10 client (latest versions) I'm testing with and both fail.

Issue:
I've setup External JWTs + OIDC as documented in your support pages and following this video (This video helped a lot btw!). https://www.youtube.com/watch?v=8ViQHzFUj_Y

I have a valid wildcard cert for my domain listed in the Web portion of my controller as an "alt cert" and have verified it shows up (and is trusted) when I access the controller via ziti.mydomain.net:8441/. I am adding on windows by going to Add Identity > JWT > selecting my network JWT I downloaded from ZAC and the I need to "STOP" and "Start" the Ziti connection for it to show up, but no IDP link is available or services:

Client Log when adding the identity (Windows - as it's easier to get the log for now) (with my domain replaced for privacy - can PM or Email if needed):

[2025-01-24T20:29:46.834Z]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
[2025-01-24T20:29:46.834Z]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.3.7 @g94225a3(HEAD) starting at (2025-01-24T20:29:46.834)
[2025-01-24T20:29:46.834Z]    INFO ziti-edge-tunnel:windows-scripts.c:326 remove_all_nrpt_rules() removing NRPT rules matching filter: $_.Comment.StartsWith('Added by ziti-edge-tunnel')
[2025-01-24T20:29:47.353Z]    INFO ziti-edge-tunnel:instance-config.c:72 load_tunnel_status_from_file() Loading config file from c:\windows\system32\config\systemprofile\appdata\roaming\netfoundry\config.json
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1424 run() ============================ service begins ================================
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1425 run() Logger initialization
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1427 run() 	- config file      : c:\windows\system32\config\systemprofile\appdata\roaming\netfoundry\config.json
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1429 run() 	- initialized at   : Fri Jan 24 2025, 15:29:47 PM (local time), 2025-01-24T20:29:47 (UTC)
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1430 run() 	- log file location: C:\Program Files (x86)\NetFoundry Inc\Ziti Desktop Edge\logs\service\ziti-tunneler.log.202501240000.log
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1432 run() 	- C SDK Version    : 1.3.7:HEAD@g94225a3
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1433 run() 	- Tunneler SDK     : v1.3.9
[2025-01-24T20:29:47.355Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1434 run() ============================================================================
[2025-01-24T20:29:47.355Z]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
[2025-01-24T20:29:47.358Z]    INFO ziti-edge-tunnel:tun.c:195 tun_open() Wintun v0.0 loaded
[2025-01-24T20:29:47.358Z]    INFO ziti-edge-tunnel:tun.c:166 flush_dns() DnsFlushResolverCache succeeded
[2025-01-24T20:29:47.441Z]    INFO ziti-edge-tunnel:tun.c:98 WintunLogger() Using existing driver 0.14
[2025-01-24T20:29:47.450Z]    INFO ziti-edge-tunnel:tun.c:98 WintunLogger() Creating adapter
[2025-01-24T20:29:47.647Z]    INFO ziti-edge-tunnel:tun.c:449 if_change_cb() default route is now via if_idx[21]
[2025-01-24T20:29:47.647Z]    INFO ziti-edge-tunnel:tun.c:455 if_change_cb() updating excluded routes
[2025-01-24T20:29:47.737Z]    INFO ziti-edge-tunnel:tun.c:98 WintunLogger() Removed orphaned adapter "ziti-tun0 1"
[2025-01-24T20:29:48.997Z]    INFO ziti-edge-tunnel:windows-scripts.c:491 is_nrpt_policies_effective() NRPT policies are effective in this system
[2025-01-24T20:29:49.515Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:865 run_tunnel() Setting interface metric to 255
[2025-01-24T20:29:49.522Z]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.3.9)
[2025-01-24T20:29:49.526Z]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
[2025-01-24T20:29:49.526Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1027 run_tunneler_loop() Loading identity files from C:\WINDOWS\system32\config\systemprofile\AppData\Roaming\NetFoundry
[2025-01-24T20:29:49.526Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:403 load_identities() loading identity file: ziti.domain.net.json
[2025-01-24T20:29:49.534Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1163 load_ziti_async() attempting to load ziti instance[c:\windows\system32\config\systemprofile\appdata\roaming\netfoundry\ziti.domain.net.json]
[2025-01-24T20:29:49.534Z]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1170 load_ziti_async() loading ziti instance[c:\windows\system32\config\systemprofile\appdata\roaming\netfoundry\ziti.domain.net.json]
[2025-01-24T20:29:49.534Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:420 load_id_cb() identity[c:\windows\system32\config\systemprofile\appdata\roaming\netfoundry\ziti.domain.net.json] loaded
[2025-01-24T20:29:49.534Z]    INFO ziti-sdk:ziti.c:505 ziti_start_internal() ztx[0] enabling Ziti Context
[2025-01-24T20:29:49.539Z]    INFO ziti-sdk:ziti.c:522 ziti_start_internal() ztx[0] using tlsuv[v0.33.4/OpenSSL 3.3.1 4 Jun 2024]
[2025-01-24T20:29:49.539Z]    INFO ziti-sdk:ziti_ctrl.c:632 ziti_ctrl_init() ctrl[(null):] using https://ziti.domain.net:8441/
[2025-01-24T20:29:49.539Z]    INFO ziti-sdk:ziti.c:600 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti.domain.net:8441/]
[2025-01-24T20:29:49.556Z]   ERROR tlsuv:engine.c:923 openssl: handshake was terminated: error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:49.556Z]   ERROR tlsuv:tls_link.c:113 TLS(000002cdba9056b0) handshake error error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:49.556Z]   ERROR tlsuv:http.c:189 handshake failed status[3]: error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:49.556Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:29:49.556Z]    WARN ziti-sdk:ziti_ctrl.c:342 internal_version_cb() ctrl[ziti.domain.net:8441] CONTROLLER_UNAVAILABLE(software caused connection abort)
[2025-01-24T20:29:49.556Z]    WARN ziti-sdk:ziti.c:1908 version_pre_auth_cb() ztx[0] failed to get controller version: CONTROLLER_UNAVAILABLE/software caused connection abort
[2025-01-24T20:29:49.556Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:29:49.556Z]    INFO ziti-sdk:ziti_ctrl.c:187 ctrl_resp_cb() ctrl[ziti.domain.net:8441] attempting to switch endpoint
[2025-01-24T20:29:49.556Z]    WARN ziti-sdk:ziti_ctrl.c:605 ctrl_next_ep() ctrl[ziti.domain.net:8441] no controllers are online
[2025-01-24T20:29:49.556Z]    WARN ziti-sdk:ziti.c:641 ext_jwt_singers_cb() ztx[0] failed to get external auth providers: software caused connection abort
[2025-01-24T20:29:54.563Z]   ERROR tlsuv:engine.c:923 openssl: handshake was terminated: error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:54.563Z]   ERROR tlsuv:tls_link.c:113 TLS(000002cdba9056b0) handshake error error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:54.563Z]   ERROR tlsuv:http.c:189 handshake failed status[3]: error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:54.563Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:29:54.563Z]    WARN ziti-sdk:ziti_ctrl.c:342 internal_version_cb() ctrl[ziti.domain.net:8441] CONTROLLER_UNAVAILABLE(software caused connection abort)
[2025-01-24T20:29:54.563Z]    WARN ziti-sdk:ziti.c:1908 version_pre_auth_cb() ztx[0] failed to get controller version: CONTROLLER_UNAVAILABLE/software caused connection abort
[2025-01-24T20:29:54.563Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:29:54.563Z]    INFO ziti-sdk:ziti_ctrl.c:187 ctrl_resp_cb() ctrl[ziti.domain.net:8441] attempting to switch endpoint
[2025-01-24T20:29:54.563Z]    WARN ziti-sdk:ziti_ctrl.c:605 ctrl_next_ep() ctrl[ziti.domain.net:8441] no controllers are online
[2025-01-24T20:29:54.563Z]    WARN ziti-sdk:ziti.c:641 ext_jwt_singers_cb() ztx[0] failed to get external auth providers: software caused connection abort
[2025-01-24T20:29:59.693Z]   ERROR tlsuv:engine.c:923 openssl: handshake was terminated: error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:59.693Z]   ERROR tlsuv:tls_link.c:113 TLS(000002cdba9056b0) handshake error error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:59.693Z]   ERROR tlsuv:http.c:189 handshake failed status[3]: error:00000005:lib(0)::reason(5)
[2025-01-24T20:29:59.693Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:29:59.693Z]    WARN ziti-sdk:ziti_ctrl.c:342 internal_version_cb() ctrl[ziti.domain.net:8441] CONTROLLER_UNAVAILABLE(software caused connection abort)
[2025-01-24T20:29:59.693Z]    WARN ziti-sdk:ziti.c:1908 version_pre_auth_cb() ztx[0] failed to get controller version: CONTROLLER_UNAVAILABLE/software caused connection abort
[2025-01-24T20:29:59.693Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:29:59.693Z]    INFO ziti-sdk:ziti_ctrl.c:187 ctrl_resp_cb() ctrl[ziti.domain.net:8441] attempting to switch endpoint
[2025-01-24T20:29:59.693Z]    WARN ziti-sdk:ziti_ctrl.c:605 ctrl_next_ep() ctrl[ziti.domain.net:8441] no controllers are online
[2025-01-24T20:29:59.693Z]    WARN ziti-sdk:ziti.c:641 ext_jwt_singers_cb() ztx[0] failed to get external auth providers: software caused connection abort
[2025-01-24T20:30:04.701Z]   ERROR tlsuv:engine.c:923 openssl: handshake was terminated: error:00000005:lib(0)::reason(5)
[2025-01-24T20:30:04.701Z]   ERROR tlsuv:tls_link.c:113 TLS(000002cdba9056b0) handshake error error:00000005:lib(0)::reason(5)
[2025-01-24T20:30:04.701Z]   ERROR tlsuv:http.c:189 handshake failed status[3]: error:00000005:lib(0)::reason(5)
[2025-01-24T20:30:04.701Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:30:04.701Z]    WARN ziti-sdk:ziti_ctrl.c:342 internal_version_cb() ctrl[ziti.domain.net:8441] CONTROLLER_UNAVAILABLE(software caused connection abort)
[2025-01-24T20:30:04.701Z]    WARN ziti-sdk:ziti.c:1908 version_pre_auth_cb() ztx[0] failed to get controller version: CONTROLLER_UNAVAILABLE/software caused connection abort
[2025-01-24T20:30:04.701Z]    WARN ziti-sdk:ziti_ctrl.c:184 ctrl_resp_cb() ctrl[ziti.domain.net:8441] request failed: -4079(software caused connection abort)
[2025-01-24T20:30:04.701Z]    INFO ziti-sdk:ziti_ctrl.c:187 ctrl_resp_cb() ctrl[ziti.domain.net:8441] attempting to switch endpoint
[2025-01-24T20:30:04.701Z]    WARN ziti-sdk:ziti_ctrl.c:605 ctrl_next_ep() ctrl[ziti.domain.net:8441] no controllers are online
[2025-01-24T20:30:04.701Z]    WARN ziti-sdk:ziti.c:641 ext_jwt_singers_cb() ztx[0] failed to get external auth providers: software caused connection abort

Controller Logs:

Jan 24 20:29:29 HOME1ZITI ziti[43169]: {"_context":"tls:0.0.0.0:8441","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.153/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.1:51273","time":"2025-01-24T20:29:29.891Z"}
Jan 24 20:29:34 HOME1ZITI ziti[43169]: {"_context":"tls:0.0.0.0:8441","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.153/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.1:56551","time":"2025-01-24T20:29:34.913Z"}
Jan 24 20:29:39 HOME1ZITI ziti[43169]: {"_context":"tls:0.0.0.0:8441","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.153/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.1:17473","time":"2025-01-24T20:29:39.935Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.153/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.1:61369","time":"2025-01-24T20:29:45.523Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"_context":"ch{ab1igzOdF}-\u003eu{classic}-\u003ei{1NRz}","file":"github.com/openziti/ziti/controller/handler_ctrl/close.go:49","func":"github.com/openziti/ziti/controller/handler_ctrl.(*xctrlCloseHandler).HandleClose","level":"info","msg":"closing Xctrl instances","time":"2025-01-24T20:29:45.679Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/handler_ctrl/close.go:36","func":"github.com/openziti/ziti/controller/handler_ctrl.(*closeHandler).HandleClose","level":"warning","msg":"disconnected","routerId":"ab1igzOdF","time":"2025-01-24T20:29:45.679Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"connected":false,"file":"github.com/openziti/ziti/controller/network/router_messaging.go:506","func":"github.com/openziti/ziti/controller/network.(*routerChangedEvent).handle","level":"info","msg":"calculating router updates for router","routerId":"ab1igzOdF","time":"2025-01-24T20:29:45.680Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/env/broker.go:139","func":"github.com/openziti/ziti/controller/env.(*Broker).RouterDisconnected.func1","level":"info","msg":"broker detected router with id ab1igzOdF disconnecting","routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","time":"2025-01-24T20:29:45.680Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/sync_strats/sync_instant.go:354","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).RouterDisconnected","level":"info","msg":"edge router [ab1igzOdF] disconnect event, router rtx removed","routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","sync_strategy":"instant","time":"2025-01-24T20:29:45.680Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"arch":"amd64","buildDate":"2024-11-23T00:09:04Z","file":"github.com/openziti/ziti/controller/handler_ctrl/accept.go:128","func":"github.com/openziti/ziti/controller/handler_ctrl.(*CtrlAccepter).Bind","level":"info","msg":"accepted new router connection","os":"linux","revision":"9a83ca87bc5f","routerId":"ab1igzOdF","time":"2025-01-24T20:29:45.763Z","version":"v1.2.2"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"connected":true,"file":"github.com/openziti/ziti/controller/network/router_messaging.go:506","func":"github.com/openziti/ziti/controller/network.(*routerChangedEvent).handle","level":"info","msg":"calculating router updates for router","routerId":"ab1igzOdF","time":"2025-01-24T20:29:45.763Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/network/router_messaging.go:312","func":"github.com/openziti/ziti/controller/network.(*RouterMessaging).sendTerminatorValidationRequest","level":"info","msg":"queuing validate of terminator","terminatorId":"2LcqgE5Cl8WVWsAZi6l4JZ","time":"2025-01-24T20:29:45.763Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/network/router_messaging.go:312","func":"github.com/openziti/ziti/controller/network.(*RouterMessaging).sendTerminatorValidationRequest","level":"info","msg":"queuing validate of terminator","terminatorId":"2QFIEFIjkg9g8yt2iod2PS","time":"2025-01-24T20:29:45.763Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/env/broker.go:125","func":"github.com/openziti/ziti/controller/env.(*Broker).RouterConnected.func1","level":"info","msg":"broker detected edge router with id ab1igzOdF connecting","routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","time":"2025-01-24T20:29:45.763Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/sync_strats/sync_instant.go:329","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).RouterConnected","level":"info","msg":"edge router connected, adding to sync routerConnectedQueue","routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","syncStatus":"SYNC_QUEUED","sync_strategy":"instant","time":"2025-01-24T20:29:45.763Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/sync_strats/sync_instant.go:487","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).hello","level":"info","msg":"edge router sync starting","routerChannelIsOpen":true,"routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","routerTxId":"Y8yqVNct3","strategy":"instant","time":"2025-01-24T20:29:45.764Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"file":"github.com/openziti/ziti/controller/sync_strats/sync_instant.go:496","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).hello","level":"info","msg":"sending edge router hello","routerChannelIsOpen":true,"routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","routerTxId":"Y8yqVNct3","strategy":"instant","syncStatus":"SYNC_HELLO","time":"2025-01-24T20:29:45.764Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"arch":"amd64","buildDate":"2024-11-23T00:09:04Z","data":null,"file":"github.com/openziti/ziti/controller/sync_strats/sync_instant.go:644","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).ReceiveClientHello","level":"info","listeners":[{"address":{"value":"tls:0.0.0.0:8442","protocol":"tls","hostname":"0.0.0.0","port":8442},"advertise":{"value":"ziti.domain.net:8442","protocol":"tls","hostname":"ziti.domain.net","port":8442}}],"msg":"edge router sent hello with version [v1.2.2] to controller with version [v1.2.2]","os":"linux","protocolPorts":["8442"],"protocols":["tls"],"revision":"9a83ca87bc5f","routerChannelIsOpen":true,"routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","routerTxId":"Y8yqVNct3","strategy":"instant","time":"2025-01-24T20:29:45.766Z","version":"v1.2.2"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"SupportsRouterModel":false,"file":"github.com/openziti/ziti/controller/sync_strats/sync_instant.go:660","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).synchronize","level":"info","msg":"started synchronizing edge router","routerChannelIsOpen":true,"routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","routerTxId":"Y8yqVNct3","strategy":"instant","time":"2025-01-24T20:29:45.766Z"}
Jan 24 20:29:45 HOME1ZITI ziti[43169]: {"SupportsRouterModel":false,"file":"github.com/openziti/ziti/controller/sync_strats/sync_instant.go:650","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).synchronize.func1","level":"info","msg":"exiting synchronization, final status: SYNC_DONE","routerChannelIsOpen":true,"routerFingerprint":"ed95ba0d60c941d1b2b6215e802a1a404525ce98","routerId":"ab1igzOdF","routerName":"HOME1ZITI-edge-router","routerTxId":"Y8yqVNct3","strategy":"instant","time":"2025-01-24T20:29:45.766Z"}
Jan 24 20:29:49 HOME1ZITI ziti[43169]: {"_context":"tls:0.0.0.0:8441","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.153/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.1:31120","time":"2025-01-24T20:29:49.824Z"}
Jan 24 20:29:54 HOME1ZITI ziti[43169]: {"_context":"tls:0.0.0.0:8441","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.153/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.1:44983","time":"2025-01-24T20:29:54.831Z"}
Jan 24 20:29:59 HOME1ZITI ziti[43169]: {"_context":"tls:0.0.0.0:8441","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.153/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.1:55172","time":"2025-01-24T20:29:59.961Z"}

Hi @007bond007,

Thanks for DM'ing me your network's jwt. I connected to your controller using openssl

openssl s_client -connect your.controller:8441 </dev/null | openssl x509 -text

Looking at the X509v3 Subject Alternative Name: section from that result, I see it's a wildcard url signed by Lets Encrypt. That's the problem.

The controller cannot be configured to 'advertise' the same url as the lets encrypt certificate..

When you setup the controller originally, did you use a name that overlaps with your wildcard cert? I think you have.

I would say:

This will prevent the private PKI from overlapping your LE cert which is like: *.clint.net

That make sense?

1 Like

Ahh ok. I think it does. To clarify... Yes I used ziti.domain.net for everything ziti related (so controller/router and Zac). I access the Zac with that domain (going through haproxy first for SSL termination) and on the appropriate ports the controller/router is accessible.

It makes sense that the overlap in what the cert covers would be an issue given it uses SNI, I totally missed that! Does each component need to have a unique domain name in a seperate SSL context then? If. I were to add another router could the configuration be as below?:

Controller:
Controller.ziti.domain.net

Router (inside my network):
Router1.ziti.domain.net

Router (hosted in the cloud):
Router2.ziti.domain.net

ZAC:
Ziti.domain.net (using that wildcard LE cert?)

Does that make sense?

Thank-you very much for looking at this for me. :slight_smile:

No. It's vital in only two/three situations I can think of (two come to mind immediately, there might be others I'm forgetting) when using alt-server certs. Any time you decide to use an alt-server cert, you must ensure the alt-server cert does not overlap with the component's "advertise" address or SNI is indeterminate.

The two times I can think of are:

  • the OpenZiti controller APIs (management/client/oidc)
  • the OpenZiti controller with the /zac binding (to deliver the ZAC via a url that doesn't give people the scary untrusted warning)
  • a router configured with web socket bindings to support BrowZer

In all of these situations an HTTP listener is desired to provide endpoints pre-configured with trust via the OS truststore.

Well, different ports are still the same domain name... So "no" to that. Recently I discussed something related with HAproxy on this discourse post. There's an associated video on that post too and github repo you might find value in. Each component is it's own server and OpenZiti will not tolerate TLS being terminated anywhere BUT that component, so each component will have it's own TLS context.

The four domains you listed would be perfect, yes. Controller.ziti.domain.net, router1.ziti.domain.net, router2.ziti.domain.net would all use the private PKI established when creating the OpenZiti overlay and zac.domain.net would use LE (matching *.domain.net only) and would be fine. You'll ALSO be able to access the zac from controller.ziti.domain.net/zac if you wanted to. Are you deploying the ZAC as part of the controller? I only ask because you might just end up using "https://ziti.domain.net/zac" or something like if you do (not zac.domain.net)

Hopfeully all that's clear :slight_smile:

1 Like

Thank-you very much! I'll rebuild and report back :slight_smile:

Ok, this is all working now... thank-you for your help! I have a few additional questions/feedback around this though:

  • If I disable/enable the connection while also logging out of Authentik in-between, I've had it fail where I can no longer get the Auth link to the IDP on Android. I can try and repeat and send logs if this helps.
  • I noticed that if I authenticate with the Android or Windows tunneler, and then log out of Authentik in a browser, I'm still authenticated with Ziti - is there a "time out" or does it re-verify the auth with the IDP occasionally (I believe other solutions utilize the IDPs timeout values... e.g. every 5 minutes if you try to turn on the VPN tunnel, you will be directed to the IDP again (which if you're logged in, happens very quickly and it just auto-closes). This is important for ZTNA/Security so if a user is no longer active or removed from a group membership granting them permission, it's caught quickly.
  • Are there plans to make it easier to login? If I look at the experience with NetBird, you click the big circle to connect, and then the IDP page opens automatically and if you're already signed in the browser window closes quickly and you're authenticated - very user friendly. With Ziti, on Android you need to enroll using the network JWT, sometimes disable/re-enable the identity/ziti tunnel, click on the identity, click on the Authenticate link (which isn't an obvious link) and then select the IDP, and then sign in/wait for auth with the IDP to happen, and then manually close the browser. This is quite a lot of steps.

Yeah, clear steps to reproduce would be very helpful and appreciated!

This is how things work for now. There are a myriad of reasons for this behavior, but this is expected at this time. In the future, there will likely be some mechanism for OpenZiti to realize the authentication has expired/is no longer valid.

If you are connected to OpenZiti, you will remain connected. If you disconnect from the overlay (toggle the identity off, turn off a tunneler etc) then you'll have to reauthenticate. If you merely disrupt the network, the session remains valid for a default period of 30m. If you are offlne for > 30 minutes, when the tunneler wakes back up you'll have to re-auth.

Specifically, this is an Android question, right? The windows tunneler already let's you "add by url". I expect that will be added at some point. We have to prioritize what we work on and when. So I expect it'll be there eventually.

Yeah, clear steps to reproduce would be very helpful and appreciated!

I'll let you know - I think I'm a bit confused how identities apply when using a network JWT - typically you have an identity for each device when using certs, but with JWT it seems like it is per user (because of how IDPs work) and maybe per device as well - I seem to have to re-add the identity when switching between Windows and Android.

If you are connected to OpenZiti, you will remain connected. If you disconnect from the overlay (toggle the identity off, turn off a tunneler etc) then you'll have to reauthenticate. If you merely disrupt the network, the session remains valid for a default period of 30m. If you are offlne for > 30 minutes, when the tunneler wakes back up you'll have to re-auth.

Thankyou for clarifying - I didn't see this documented anywhere :slight_smile: I would suggest, given the nature of Zero-Trust that the user's session is re-validated every X minutes to ensure it remains valid... always verify.

Specifically, this is an Android question, right? The windows tunneler already let's you "add by url". I expect that will be added at some point. We have to prioritize what we work on and when. So I expect it'll be there eventually.

I think it's really a user experience issue across both Android and Windows tunnelers, although it's worse on Android for sure. The number of items that need to be accomplished just to get to the IDP login page seems excessive.

Some recommendations on my testing so far:

  • When turning on/adding an identity have the IDP link automatically open if there's only one IDP available

  • If there's multiple IDPs, allow setting (In ZAC) the default IDP to authenticate with, client could override this if they have access to other IDPs (though, I don't know why anyone would use multiple IDPs)

  • Allow enrolling via URL on Android

  • Resolve the browser not closing after IDP auth. I know you mention in your video that it's only the first time you access it that it does not close, but I've found that it never has closed for me on Windows 10 or Android on my Pixel 8. If you perhaps look at the code NetBird uses for this, maybe it could be adapted for Ziti as it works every time.

FWIW - the windows client should be doing all these things. If that's not the case, I'd like to try to reproduce what you're seeing. I worked on the windows client, but not the android one.

You're saying the netbird experience isn't the same? Maybe I should go through that experience. I've seen plenty of "you can close this window" pages in my days, maybe there's something else going on...

1 Like

FWIW I went through the whole process and they seem to have the same issue we do on their "you can close this" page so -- I'm happy it works for you but this stuck around for me:

Very strange - just tried it today to see if it was the same result and NetBird closes and goes straight back to the app, where Ziti does not. I noticed that it uses a different type of browser window though, not my default Chrome browser on Android - perhaps that's how they get around it if the specific phone supports it?

Looking at the javascript that closes the window and some online resources, it seems like it may be a browser security feature interfering because the javascript didn't initiate the window, so it cannot close it. Perhaps there's a way to launch the browser that would allow it to work.


As for the Windows and Android user experience feedback - In your video the Windows client is a little better but does not automatically open the IDP when enabling the identity (I just confirmed in the video here).

In Android you need to click deeper by going into the details and clicking on the Authentication link and then selecting the IDP (even if there's only one) and then authenticate with the IDP.

An improved flow would be like this:
User opens Ziti Tunneler app > Clicks enable on the identity > IDP page automatically opens > User authenticates and the page closes - for both Android and Windows (and other clients in the future).

Yeah, I haven't found the magic sauce that is 100% reliable wrt browsers allowing javascript to close an OIDC flow. I wish I could find that magic tho. You might be doing some different flow though. Could you share 'how' you did that? I want to look at their javascript to see how far/close it is to what i came up with.

yeah it won't auto-auth you when you add the identity. you still need to click the icon. i don't think that'll end up changing tbh. not yet anyway :slight_smile:

It works on Android, but not Windows it seems (For NetBird). I'm using the latest self hosted version of NetBird and latest clients, and using Authentik as the IDP. Nothing special there other than those specific items - followed the guides provided on their self hosting documentation.

yeah it won't auto-auth you when you add the identity. you still need to click the icon. i don't think that'll end up changing tbh. not yet anyway :slight_smile:

No problem, just providing feedback to provide a nicer user experience. Less clicks = happier users typically. The Android client is where the most improvement can be I feel.

So, it might surprise you but I've literally never used/deployed netbird, so I have no idea what I'm doing there... :slight_smile: In fact, I didn't even realize they HAD a self-hosted version until you said they did. Somehow I missed that... So what I was hoping for was just a short list of do this do that's... I don't think I'm doing it how you did it. Sounds like it's:

  • install netbird "self-hosted" Advanced guide - NetBird Docs
  • install netbird windows client
  • authenticate windows client to netbird (? <-- is THIS when the idp popup appears and goes away?)
  • do stuff... go away...
  • come back to windows and have to re authenticate(? <-- is THIS when the idp popup appears and goes away?)

I'll try it out, but if you can tell me your flow, that'll help me be efficient with my time. :slight_smile:

And, much appreciated on the feedback fwiw, thanks!

Yes the advanced install guide and the part with an IDP instead of using their default one, I set it up with Authentik for the IDP. To clarify as mentioned previously, the windows client does NOT auto-close the authenticated page, but Android DOES. It's all good though - that's a lot of work just to sort out Android's auto-close feature.

But yes authenticating on ANDROID the auth window auto-closes itself when initiating the connection to NetBird. (so your first ?)

No problem with regards to the feedback - I think OpenZiti has a ton of potential both for personal homelab use and business use, it just needs that little more user-friendliness for more to consider it/understand it.