Controller & Router Helm | Windows Edge Tun Can't Auth with Ext-jwt-signer

Hello,

After successfully deploying a Ziti Controller & Router using Helm with an alternative issuer set (Hashicorp Vault) I've been working on trying to get the Windows Edge Client to connect and authenticate with an External JWT Signer (Keycloak).

I've run into many issues along the way, but for the sake of this post I will focus on purely authenticating with an Ext JWT Signer (keycloak) against the ZAC UI, and with the Windows Edge Client.

Forgive my ignorance as I navigate, a few concepts are still new to me. The helm deployment is quite a bit more complex than the express install I played around with last year.

Starting with the ZAC and logging in with an OIDC provider, I am not sure why the option is not appearing on the login page under the generic username + password form. I've tried to follow the following doc -> Keycloak | OpenZiti. I might be missing a policy to have this appear on the ZAC gui and am looking for guidance.

And I believe the actual OIDC login flow works as I see the events on keycloak with a JWT returned (when testing with the Windows Edge Client). My external identity shows as connected for the API session indicator, however Edge Router Connected indicator is not green. When I click on the "authorize idp" button on the windows client, I get the browser popup and can complete the flow with a successful localhost call back page. As well as everything looking successful as far as Keycloak is concerned.

Though after the localhost page closes, the ziti desktop client shows Auth Request Failed. Checking the logs I see the following

[2025-05-02T18:54:05.199Z]   ERROR ziti-sdk:ziti_ctrl.c:522 ctrl_body_cb() ctrl[<redacted>:443/] API request[/current-identity/edge-routers?limit=25&offset=0] failed code[UNAUTHORIZED] message[The request could not be completed. The session is not authorized or the credentials are invalid]
[2025-05-02T18:54:05.199Z]   ERROR ziti-sdk:ziti.c:1475 edge_routers_cb() ztx[0] failed to get current edge routers: code[401] UNAUTHORIZED/The request could not be completed. The session is not authorized or the credentials are invalid
[2025-05-02T18:54:05.231Z]   ERROR ziti-sdk:ziti_ctrl.c:522 ctrl_body_cb() ctrl[<redacted>:443/] API request[/current-api-session/service-updates] failed code[UNAUTHORIZED] message[The request could not be completed. The session is not authorized or the credentials are invalid]
[2025-05-02T18:54:05.231Z]    WARN ziti-sdk:ziti.c:1423 check_service_update() ztx[0] failed to poll service updates: code[401] err[-14/The request could not be completed. The session is not authorized or the credentials are invalid]
[2025-05-02T18:55:56.823Z]   ERROR ziti-sdk:ziti.c:1550 update_identity_data() ztx[0] failed to get identity_data: The request could not be completed. The session is not authorized or the credentials are invalid[UNAUTHORIZED]

When checking the controller pod logs I see a bunch of the following

{"authMethod":"ext-jwt","file":"github.com/openziti/ziti/controller/model/authenticator_mod_ext_jwt.go:372","func":"github.com/openziti/ziti/controller/model.(*AuthModuleExtJwt).process","level":"error","msg":"encountered 0 candidate JWTs, verification cannot occur","time":"2025-05-02T18:57:14.889Z"}
{"authMethod":"ext-jwt","file":"github.com/openziti/ziti/controller/model/authenticator_mod_ext_jwt.go:372","func":"github.com/openziti/ziti/controller/model.(*AuthModuleExtJwt).process","level":"error","msg":"encountered 0 candidate JWTs, verification cannot occur","time":"2025-05-02T18:57:14.911Z"}

From searching through forum posts I found that I might need to create a router policy. I tried the steps listed in this post -> New install help - ziti edge router is not available - openziti

# Allow all identities to use any edge router with the "public" attribute
ziti edge create edge-router-policy all-endpoints-public-routers --edge-router-roles "#public" --identity-roles "#all"

# Allow all edge-routers to access all services
ziti edge create service-edge-router-policy all-routers-all-services --edge-router-roles "#all" --service-roles "#all"

But consequently, it did not help, additionally, access to the Kube API was lost and I had to delete the policies for it to be restored. Which shows I still do not fully understand the implications of running ziti in a k8s cluster where other services are running, i'll have to do a bit more reading into that one.

If it is a router policy that I need, is there a way I can implement them without affecting other services running on the cluster?

I apologize for the long post! I'm trying to get a repeatable deployment model with K8s going vs relying on VMs and the express install. Thanks for your guidance!

1 Like

Hi @Tetrusp,

The deployment mode really shouldn't matter. With both you need to make sure you get the PKI setup correctly. I'm going to assume you've done that.

Verify Traffic

I personally use the ziti ops verify traffic command. For example, here you can see a successful run, even though the edge router thinks there's some sort of configuration issue to the router router=ip-172-31-11-231-edge-router. Even with that error, the traffic test succeeds (green)

Verify ext-jwt-signer is correct

Assuming that works, then next you can use the ziti ops verify ext-jwt-signer oidc command to see if you have the external jwt signer setup properly. Another example command:

ziti ops verify ext-jwt-signer oidc --controller-url https://ec2-3-142-245-63.us-east-2.compute.amazonaws.com:8441 browzer-keycloak-ext-jwt-signer

Here you can see I'm using keycloak. In case you're interested in what that looks like, it looks like this:

Running verify oidc

When running the verify oidc command, you should see something like somewhat similar to this output:

Enabling an Identity

Assuming you can see the token contents, you then need to make an identity and map that identity's 'externalId' to one of those claims that come back in one of the tokens (ID or Access). Often email, but it could be sub or whatever...

Success

Now add --authenticate to the verify oidc command:

ziti ops verify ext-jwt-signer oidc --controller-url https://ec2-3-142-245-63.us-east-2.compute.amazonaws.com:8441 browzer-keycloak-ext-jwt-signer --authenticate

You will need to see success, like the following.
image

Let's pause and make sure you can get this far, then we can finish it off....