HA not working as expected with ext-jwt signer (without JIT enrollment)

1

I have an 2.0.0-pre12 setup that is configured to use a local Keycloak instance as an external JWT signer without JIT enrollment of external identities. This means I'm pre-creating the identities for users in Ziti with their external ID configured to use our Keycloak, so they can simply login from any endpoint with an Edge Client installed and have their services available from wherever they are.

Now this setup works just fine with a standalone controller. But I'm currently trying out controller clustering and this setup seems to be not compatible with HA:

Clustering and Raft works fine, everything is setup and when using an identity enrolled with a JWT, all the controller failover scenarios work just fine. Edge client is picking up the new leader, reauthenticates and all my sessions to any Ziti services in the network work just fine even when stopping and starting the tunnel on the client side.

Now when I'm using a precreated identity as mentioned above that is simply configured with Keycloak as the ext-jwt-signer, this breaks. When the initial controller goes down, so that there are just two cluster members remaining and the leader changes, the currently authenticated session works just fine.

But as soon as I stop and restart the tunnel on the client side and reauth with Keycloak (this also works and I see my services in the Edge client as green) and want to connect to a service again, the router that is terminating the service is throwing invalid client certificate errors:

{"error":"invalid client certificate for api session","file":"github.com/openziti/ziti/v2/router/xgress_edge/accept.go:288","func":"github.com/openziti/ziti/v2/router/xgress_edge.(*Acceptor).handleUngroupedUnderlay","level":"error"
,"msg":"failure accepting edge channel u{classic}-\u003ei{ziti-sdk-c[0]@Mac.lan/P8Zb} with underlay","time":"2026-05-06T09:54:20.601Z"}

It seems like the fact that with this setup (which works just fine in a single-controller setup) that is missing the client certificate enrollment, HA is not working as the routers depend on a client certificate that is somehow not there with this "simple" ext-jwt-signer setup.

As mentioned, when I enroll an identity with a OTT JWT from Ziti itself, the failover works just fine, but for users that need to be able to jump between machines with their same Keycloak account, it is not really an option to re-enroll their identity every time they use a different computer.

2

We recently detected a bug where token based authentication from some SDKs would errantly include a client certificate.

Do you happen to know off hand if you used to have an certificate based enrollment for the identity in use?

3

The identities in question were only ever created the following way:

ziti edge create identity "user@example.com" --external-id "user@example.com" -a "my_attribute"

No JWT created for those. And as mentioned, they work just fine in standalone mode and also in HA mode as long as the first cluster member stays up (doesn't even need to be the leader, just reachable by the Edge client and the router).

But as soon as the first controller node is down and I restart the tunnel in the Edge client and re-authenticate with Keycloak, the router throws the "invalid client certificate" messages.

When I create an identity with ziti edge create identity failover-test -a "my_attribute" -o ./test.jwt and use the JWT to enroll instead of Keycloak auth, all the HA stuff works just fine, even after restarting the tunnel. The router is not complaining about invalid client certs in this case.