OIDC Failing at Controller Authentication Step (PKCE + Authentik SSO)

Support
1

Hello,

I'm hoping to get some insight or support on an issue I've been stuck on for over three days involving OIDC integration with OpenZiti.

Network:

  • OpenZiti Controller: Installed directly (non-Docker) on a VPS (public)
  • Ziti Routers: Several — a mix of LAN and cloud (working correctly)
  • OIDC Provider: Authentik (hosted on the LAN side, reverse proxied)

The core network is functioning flawlessly — tunnels, identities, routers, services, all configured and working as expected.

I've now looking to integrated Authentik for SSO via OIDC using PKCE flow so i can test out Browzer, and everything appears to be functioning until the final POST to the controller for authentication. At that point, the controller responds with:

API request[/authenticate] failed code[INVALID_AUTH] message[The authentication request failed]

I am getting a successful flow, and token generated according to the tunnel logs but then when ever the authentication attempts to post to the controller it fails,

From the client tunnel logs:

  • Token request completes (200 OK)
  • ID token and access token are issued
  • Tokens contain the correct iss, aud, email, sub, azp, etc.
  • Ziti correctly receives and decodes the token

Yet, authentication fails at the controller step. Here's the log snippet (with sensitive fields redacted):

[2025-03-23T22:56:45.925Z]    INFO ziti-sdk:oidc.c:411 request_token() requesting token path[https://<AUTHENTIK_HOST>/application/o/token/] auth[<REDACTED_AUTH_CODE>]
[2025-03-23T22:56:46.306Z]   DEBUG ziti-sdk:oidc.c:395 token_cb() 200 OK
[2025-03-23T22:56:46.306Z]   DEBUG ziti-sdk:oidc.c:902 oidc_client_set_tokens() using access_token={
  "iss": "https://<AUTHENTIK_HOST>/application/o/openziti-api/",
  "sub": "<REDACTED_SUBJECT_ID>",
  "aud": "authentik_openziti",
  "exp": 1742770910,
  "iat": 1742770610,
  "auth_time": 1742759485,
  "acr": "goauthentik.io/providers/oauth2/default",
  "email": "user@example.com",
  "email_verified": true,
  "name": "John Doe",
  "given_name": "John",
  "preferred_username": "john.doe",
  "nickname": "john.doe",
  "groups": ["authentik Admins", "web backend"],
  "azp": "authentik_openziti",
  "uid": "<REDACTED_UID>"
}
[2025-03-23T22:56:46.306Z]   DEBUG ziti-sdk:external_auth.c:90 ext_token_cb() received access token: eyJhbGciOiJSUzI1NiIs...<REDACTED>
[2025-03-23T22:56:46.306Z]   DEBUG ziti-sdk:oidc.c:913 oidc_client_set_tokens() scheduling token refresh in 300 seconds
[2025-03-23T22:56:46.444Z]   DEBUG ziti-sdk:ziti_ctrl.c:499 ctrl_body_cb() ctrl[https://<ZITI_CONTROLLER_HOST>:1280] completed POST[/authenticate] in 0.135 s
[2025-03-23T22:56:46.444Z]   ERROR ziti-sdk:ziti_ctrl.c:521 ctrl_body_cb() ctrl[https://<ZITI_CONTROLLER_HOST>:1280] API request[/authenticate] failed code[INVALID_AUTH] message[The authentication request failed]
[2025-03-23T22:56:46.444Z]   ERROR ziti-sdk:ziti_ctrl.c:388 ctrl_login_cb() ctrl[https://<ZITI_CONTROLLER_HOST>:1280] INVALID_AUTH(The authentication request failed)
[2025-03-23T22:56:46.444Z]   DEBUG ziti-sdk:ziti_ctrl.c:379 ziti_ctrl_clear_api_session() ctrl[https://<ZITI_CONTROLLER_HOST>:1280] clearing api session token for ziti_controller
[2025-03-23T22:56:46.444Z]    WARN ziti-sdk:legacy_auth.c:183 login_cb() failed to login to ctrl[https://<ZITI_CONTROLLER_HOST>:1280/] INVALID_AUTH[-14] The authentication request failed
[2025-03-23T22:56:46.444Z]   DEBUG ziti-sdk:ziti.c:266 ziti_set_impossible_to_authenticate() ztx[0] setting api_session_state[0] to 4
[2025-03-23T22:56:46.444Z]   DEBUG ziti-sdk:ziti_ctrl.c:379 ziti_ctrl_clear_api_session() ctrl[https://<ZITI_CONTROLLER_HOST>:1280] clearing api session token for ziti_controller
[2025-03-23T22:56:46.444Z]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:1004 on_ziti_event() ziti_ctx controller connections failed: failed to authenticate
[2025-03-23T22:56:46.444Z]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:456 on_event() ztx[C:\...netfoundry\<REDACTED_CONFIG>.json] context event : status is failed to authenticate
[2025-03-23T22:56:46.444Z]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:510 on_event() ztx[C:\...netfoundry\<REDACTED_CONFIG>.json] failed to connect to controller due to failed to authenticate

What I’ve Confirmed or Tried

  • iss, aud, and jwks_uri are exact matches (copied from .well-known/openid-configuration)
  • JWT signer and Auth Policy are correctly registered and show in the controller (ziti edge list)
  • Tried using email and sub as the claimsProperty, mapped to an externalId on the identity
  • Verified the identity exists and is assigned the correct Auth Policy
  • Tried with:
  • existing identity (matching external ID)
  • new identity creation via OIDC
  • Token decodes successfully with all expected fields (via JWT.io)
  • Tried with public and confidential client modes
  • Controller logging is enabled (debug), but nothing relevant is logged for this failure
  • Controllers domain is verified via third-party cert, and is not using the same as the advertised address.

The fact that Ziti SDK decodes the token, logs it, i get redirected to a successful callback and proceeds to call /authenticate, but still gets a INVALID_AUTH with no additional context from the controller, makes this difficult to debug.

I've tried re-creating everything, i've checked outside of Zac, directly on the controller to ensure the auth policies and the JWT provider have been registered and they have done. Decoded the tokens to ensure the payloads contain the rellevant data.

Unfortunatly out of ideas on where else to troubleshoot this one.

  1. Are there any additional controller log levels or flags to enable deeper OIDC-specific debug output?
  2. Is there any known edge case where Ziti decodes a token but fails to authenticate even when claimsProperty matches externalId?
  3. Could this be a scope or claim mapping issue even though email and sub are included in the JWT and policy?
  4. Can I manually validate the token or debug the controller validation logic?
  5. Are there any differences in controller behavior depending on whether identities are created automatically vs pre-created?
  6. Does any part of the flow need to publically access the JWT endpoint?

I’ve reviewed the updated docs here:
https://ziti-doc-git-issue-1022using-idps-openziti.vercel.app/docs/guides/external-auth/identity-providers/authentik

And implemented everything step-by-step.

Any help would be hugely appreciated!

Thanks,
C

2

Hi @Curt_Ziti, welcome to the community and to OpenZiti!

Sounds like you discovered the myriad of IdP related doc, that's great to see, but it sounds like you're just getting stuck at the step.

The things things I've seen cause issues:

  • the issuer has a trailing slash, but the issuer in ziti omits the trailing slash
  • the audience specified is not the audience mapped on the ext-jwt-signer
  • an external id mismatch
  • jwks url is not verifiable by the OpenZiti Controller If Authentik uses a self signed cert, this would be a problem.

Are you able to authenticate using the ZAC itself? Can we start with trying the ZAC and getting that to allow for ext-jwt-signer auth? Make sure your externally mapped id is set to be an admin and try getting the ZAC to let you authenticate. Let's see if we can debug it better from the ZAC.

Also the controller usually has some logs that are actually useful when this fails. It seems like it's not useful, but could you please try authenticating and sharing the controller's error when the auth attempt occurs?

1 Like
3

I realize I didn't answer these, so I will:

Is there any known edge case where Ziti decodes a token but fails to authenticate even when claimsProperty matches externalId?

I've not found one yet. casing is important though but i expect you checked/verified that

Could this be a scope or claim mapping issue even though email and sub are included in the JWT and policy?

As long as you're certain the jwt payload has the email claim, no.

Can I manually validate the token or debug the controller validation logic?

The ziti CLI has a command:

ziti ops verify ext-jwt-signer oidc --controller-url ${controller-url} ${ext-jwt-signer-name}  --access-token --id-token

That will dump the raw ID or Access tokens out which you could inspect more closely. You can submit them back to the authenticate endpoint

curl -sk "$ctrl/edge/client/v1/authenticate?method=ext-jwt" \
  -X 'POST' \
  -H "Authorization: Bearer $token"

Are there any differences in controller behavior depending on whether identities are created automatically vs pre-created?

None I know of, but I'm not sure what you mean. Did you use 3rd party auto CA?

1 Like
4

One final thing came to me this morning that might be a problem if it's wrong. It's possible that you have assigned a different authentication policy to the identity. If you have done that, you must ensure it allows for external jwt auth:

image
image886×341 22.8 KB

5

Hey @TheLumberjack,

Thanks for the welcome — really cool project you guys have built here! (Though it definitely took a bit to wrap my head around things :sweat_smile:)

I was finally able to get OIDC working. The tokens were being generated correctly, the config was solid, and even the JWKS endpoint was properly signed by our internal CA (which the Ziti controller VM trusts). Interestingly, after getting it working, I tested again using self-signed certs for the toke signing keys and it still worked — so that didn’t seem to be the blocker.

The root cause turned out to be that Authentik wasn’t accessible to the controller — it’s hosted on-prem and not public-facing. I realised this when trying to use SSO with ZAC and saw it failing to access the .well-known/openid-configuration endpoint.

To solve this without exposing Authentik publicly, I set up a tunnel on the controller VM with a wildcard intercept for *.authentik.net, routing it via a LAN ziti router to our internal reverse proxy. This lets the controller access Authentik’s endpoints as if it were on LAN, and everything clicked into place: OIDC auth now works in the Windows tunnel client and ZAC.

My guess is that even though the client successfully retrieved a valid token, the controller couldn’t verify it without being able to get the pub keys from JWKS endpoint, hence the failure at the authenticate stage.

Thanks again for the help and docs!

2 Likes
6

Gah. Yeah. I forgot that is a possible problem too but it's in the 'self-signed certs' region where the controller can't return the keys, just a different reason! :slight_smile: Makes total sense. Thanks for following up!

Also - cool solution! :slight_smile:

1 Like