Hello OpenZiti team,
First of all, thank you for the great project — we have been running OpenZiti in production for several services and appreciate the stability and design of the overlay. We are writing to report a regression in the OIDC flow with an external Keycloak IdP that appeared starting with ZDEW 2.9.7.2 and persists in 2.11.3.0, while 2.9.7.1 works correctly against the same controller.
Subject:
OIDC with external Keycloak IdP: post-enrollment "Controller authentication" fails with 404/405 on ZDEW 2.9.7.2 / 2.11.3.0 — works on 2.9.7.1.
Summary:
We operate OpenZiti with an external Keycloak IdP (external JWT signer, authorization code + PKCE). With ZDEW 2.9.7.1 the full OIDC flow works (enrollment, login, edge-router channel, services). With ZDEW 2.9.7.2 and 2.11.3.0 against the same controller, enrollment succeeds and the JIT identity is created, but the identity can never complete authentication afterwards: the SDK fails at the "Controller authentication" step with HTTP 404 while configuring the internal OIDC provider, then HTTP 405 "Method Not Allowed" on the authorize request. Re-authentication ("Authorize IdP") also intermittently fails with invalid_grant (Code not valid, PKCE verification failed: Code mismatch).
Environment:
- Controller: openziti v2.1.0-pre1 (465b34d38c85, 2026-07-28), single node.
- Edge router.
- IdP: Keycloak; public client openziti-client, standard flow + PKCE (S256), redirect http://localhost:20314/auth/callback; mappers: sub, email, audience(openziti-client).
- External JWT signer: keycloak-signer — issuer = realm URL, audience openziti-client, claimsProperty: email, useExternalId: true, targetToken: ACCESS, jwksEndpoint = realm certs endpoint, enrollToTokenEnabled: true, enrollToCertEnabled: true.
- Clients (same Windows hosts): ZDEW 2.9.7.1 (C SDK 1.11.8, tlsuv 0.41.1/OpenSSL 3.6.1) — works; ZDEW 2.9.7.2 — fails; ZDEW 2.11.3.0 (C SDK 1.18.7, tlsuv 0.42.4/OpenSSL 3.6.3) — fails.
Working flow (2.9.7.1):
ext-auth link → Keycloak login → external token → /enroll/token (first time) or existing identity → oidc[internal] token exchange at /oidc/oauth/token succeeds → "connected to controller" → edge-router channel up → services intercepted.
Failures (2.9.7.2 / 2.11.3.0):
- After successful enrollment (identity file written, JIT identity with externalId created), the loaded identity cannot authenticate:
oidc[internal] initializing with provider[https://########:1280/oidc]
ERROR internal_config_cb() oidc[internal] unexpected response code[404]
ERROR oidc_client_configure() oidc[internal] cannot configure while another request is in progress
WARN failed_auth_req() oidc[internal] OIDC authorization failed: Method Not Allowed
WARN ztx_set_unauthenticated() auth error: Method Not Allowed
The local auth page (localhost:20314) shows: Step "Controller authentication", Error "Method Not Allowed", together with a valid external Keycloak token. - Re-authorization intermittently fails at token exchange:
1.1 400 Bad Request
{ "error": "invalid_grant", "error_description": "Code not valid" }
{ "error": "invalid_grant", "error_description": "PKCE verification failed: Code mismatch" } - Controller journal during repeated enrollment attempts:
ERROR EnrollModuleToken.checkForExistingIdentity: duplicate identity found for external id <user>
ERROR timeoutHandler.ServeHTTP: POST /edge/client/v1/enroll/token timeout ... returning 503
Diagnostics already performed:
- GET https://#######:1280/.well-known/openid-configuration → 404; GET https://##########:1280/oidc/.well-known/openid-configuration → 200 with correct issuer/authorize/token endpoints. Internal OIDC discovery is served under /oidc.
- Keycloak access token obtained and decoded manually: iss/aud/azp/sub/email/scope correct; JWKS reachable; signature verifies.
- Windows certificate stores verified on all hosts; the failure is at HTTP level (404/405), not a TLS handshake; identical stores work with 2.9.7.1. We do not believe this is a certificate trust issue.
- Note: OIDC-enrolled identities end up with authenticators: {} — although the signer has enrollToCertEnabled: true, the SDK logs enroll_mode=token, enrolling without CSR, so no cert authenticator is ever provisioned and the client is forced into the (broken) internal OIDC login on every start.
Questions:
- Is controller 2.1.0-pre1 supported with the internal-OIDC login flow used by C SDK 1.18.x (ZDEW 2.11.3.0)? The 404 on internal OIDC configuration followed by 405 looks like an endpoint/method mismatch between SDK and controller.
- Why does the newer SDK enroll "without CSR" when enrollToCertEnabled: true, leaving the identity without any cert authenticator?
- Any known issues with PKCE state handling on re-authentication (Code not valid / Code mismatch) when multiple OIDC contexts exist inside the service process?