Controller & Router Helm | Windows Edge Tun Can't Auth with Ext-jwt-signer

1

Hello,

After successfully deploying a Ziti Controller & Router using Helm with an alternative issuer set (Hashicorp Vault) I've been working on trying to get the Windows Edge Client to connect and authenticate with an External JWT Signer (Keycloak).

I've run into many issues along the way, but for the sake of this post I will focus on purely authenticating with an Ext JWT Signer (keycloak) against the ZAC UI, and with the Windows Edge Client.

Forgive my ignorance as I navigate, a few concepts are still new to me. The helm deployment is quite a bit more complex than the express install I played around with last year.

Starting with the ZAC and logging in with an OIDC provider, I am not sure why the option is not appearing on the login page under the generic username + password form. I've tried to follow the following doc -> Keycloak | OpenZiti. I might be missing a policy to have this appear on the ZAC gui and am looking for guidance.

And I believe the actual OIDC login flow works as I see the events on keycloak with a JWT returned (when testing with the Windows Edge Client). My external identity shows as connected for the API session indicator, however Edge Router Connected indicator is not green. When I click on the "authorize idp" button on the windows client, I get the browser popup and can complete the flow with a successful localhost call back page. As well as everything looking successful as far as Keycloak is concerned.

Though after the localhost page closes, the ziti desktop client shows Auth Request Failed. Checking the logs I see the following

[2025-05-02T18:54:05.199Z]   ERROR ziti-sdk:ziti_ctrl.c:522 ctrl_body_cb() ctrl[<redacted>:443/] API request[/current-identity/edge-routers?limit=25&offset=0] failed code[UNAUTHORIZED] message[The request could not be completed. The session is not authorized or the credentials are invalid]
[2025-05-02T18:54:05.199Z]   ERROR ziti-sdk:ziti.c:1475 edge_routers_cb() ztx[0] failed to get current edge routers: code[401] UNAUTHORIZED/The request could not be completed. The session is not authorized or the credentials are invalid
[2025-05-02T18:54:05.231Z]   ERROR ziti-sdk:ziti_ctrl.c:522 ctrl_body_cb() ctrl[<redacted>:443/] API request[/current-api-session/service-updates] failed code[UNAUTHORIZED] message[The request could not be completed. The session is not authorized or the credentials are invalid]
[2025-05-02T18:54:05.231Z]    WARN ziti-sdk:ziti.c:1423 check_service_update() ztx[0] failed to poll service updates: code[401] err[-14/The request could not be completed. The session is not authorized or the credentials are invalid]
[2025-05-02T18:55:56.823Z]   ERROR ziti-sdk:ziti.c:1550 update_identity_data() ztx[0] failed to get identity_data: The request could not be completed. The session is not authorized or the credentials are invalid[UNAUTHORIZED]

When checking the controller pod logs I see a bunch of the following

{"authMethod":"ext-jwt","file":"github.com/openziti/ziti/controller/model/authenticator_mod_ext_jwt.go:372","func":"github.com/openziti/ziti/controller/model.(*AuthModuleExtJwt).process","level":"error","msg":"encountered 0 candidate JWTs, verification cannot occur","time":"2025-05-02T18:57:14.889Z"}
{"authMethod":"ext-jwt","file":"github.com/openziti/ziti/controller/model/authenticator_mod_ext_jwt.go:372","func":"github.com/openziti/ziti/controller/model.(*AuthModuleExtJwt).process","level":"error","msg":"encountered 0 candidate JWTs, verification cannot occur","time":"2025-05-02T18:57:14.911Z"}

From searching through forum posts I found that I might need to create a router policy. I tried the steps listed in this post -> New install help - ziti edge router is not available - openziti

# Allow all identities to use any edge router with the "public" attribute
ziti edge create edge-router-policy all-endpoints-public-routers --edge-router-roles "#public" --identity-roles "#all"

# Allow all edge-routers to access all services
ziti edge create service-edge-router-policy all-routers-all-services --edge-router-roles "#all" --service-roles "#all"

But consequently, it did not help, additionally, access to the Kube API was lost and I had to delete the policies for it to be restored. Which shows I still do not fully understand the implications of running ziti in a k8s cluster where other services are running, i'll have to do a bit more reading into that one.

If it is a router policy that I need, is there a way I can implement them without affecting other services running on the cluster?

I apologize for the long post! I'm trying to get a repeatable deployment model with K8s going vs relying on VMs and the express install. Thanks for your guidance!

1 Like
2

Hi @Tetrusp,

The deployment mode really shouldn't matter. With both you need to make sure you get the PKI setup correctly. I'm going to assume you've done that.

Verify Traffic

I personally use the ziti ops verify traffic command. For example, here you can see a successful run, even though the edge router thinks there's some sort of configuration issue to the router router=ip-172-31-11-231-edge-router. Even with that error, the traffic test succeeds (green)

image
image1629×493 58.5 KB

Verify ext-jwt-signer is correct

Assuming that works, then next you can use the ziti ops verify ext-jwt-signer oidc command to see if you have the external jwt signer setup properly. Another example command:

ziti ops verify ext-jwt-signer oidc --controller-url https://ec2-3-142-245-63.us-east-2.compute.amazonaws.com:8441 browzer-keycloak-ext-jwt-signer

Here you can see I'm using keycloak. In case you're interested in what that looks like, it looks like this:

image
image1120×891 85.6 KB

Running verify oidc

When running the verify oidc command, you should see something like somewhat similar to this output:

image
image1200×1232 42.2 KB

Enabling an Identity

Assuming you can see the token contents, you then need to make an identity and map that identity's 'externalId' to one of those claims that come back in one of the tokens (ID or Access). Often email, but it could be sub or whatever...

Success

Now add --authenticate to the verify oidc command:

ziti ops verify ext-jwt-signer oidc --controller-url https://ec2-3-142-245-63.us-east-2.compute.amazonaws.com:8441 browzer-keycloak-ext-jwt-signer --authenticate

You will need to see success, like the following.
image

Let's pause and make sure you can get this far, then we can finish it off....

3

Good Afternoon @TheLumberjack,

Thank you for taking the time to help me!

I have the following errors with a default install of a Ziti Controller && Router in k8s.

image
image1592×509 60.6 KB

After following steps listed here -> No_edge_routers_available - Support - openziti

ziti edge create edge-router-policy all-ids-public-ers --identity-roles '#all' --edge-router-roles '#public'

ziti edge update edge-router <router name> -a 'public'

ziti edge create service-edge-router-policy <router policy name> --service-roles '#all' --edge-router-roles '#all'

I get the following result.

image
image1211×444 41.5 KB

Also so far, there are no issues with K8s API access, so I am unsure why adding the policies last time took down the network on that cluster.

In anycase, moving onto this command

ziti ops verify ext-jwt-signer oidc --controller-url <ziti controller> <ext-jwt-signer name> --ca <ca-file.crt>

I get a timeout

image
image1575×286 29.8 KB

I am not sure why, when testing from the desktop client I was getting through the auth flow properly from Keycloak's perspective, with browser popups to the localhost callback page showing success.

image
image604×282 21.7 KB

I got a shell in the controller and can confirm the pod can reach the Keycloak server.

image
image1850×168 17.4 KB

Could you also explain why i need to specify the CA file for every command? I checked my client json and can confirm the ca is there, however if I dont specify it, the ziti cli gives me a x509 validation error.

4

The OIDC timeout error indicates the browser didn't complete the callback in time. Is it possible you're not running it from the same computer or that there's a tab open that you didn't notice?

This is unexpected. It might be a bug with the command. Can you try logging in one time? Logging in will cache the ca locall in a file. I think that is why you need to provide the CA, it's not cached locally.

5

Hi,

The command is being run from WSL on my local machine! Though my browser code flow has been broken in WSL for a while, CLI tools usually provided me with a link to manually open in a browser. After a quick google search, I found to install wslu in WSL successfully fixed the issue for ziti.

The first verify ran successfully with a browser popping up and proceeding with the Keycloak auth flow.

ziti ops verify ext-jwt-signer oidc --controller-url <ziti controller> <ext-jwt-signer name> --ca <ca-file.crt>

image
image1594×768 46.9 KB

image
image419×253 11.2 KB

The second command failed with a 401 Unauthorized.

image
image1281×270 19.5 KB

After creating a new identity on the controller for the user, the --authenticate ran successfully

image
image694×117 9.42 KB

I decided to try the desktop edge client again for fun...and its decided to just work this time?!

image
image451×519 39.1 KB

I am relieved as this has not been working for a while, I guess writing this post helped me work through the issues! However, I am very confused now as to what I might have done differently lol

In any case thanks again for your guidance!

In regards to the CLI and trying to login again, I get the same behavior of requiring to specify a CA file

6

You fixed the NO_EDGE_ROUTERS problem. I suppose it's possible that something you did ended up fixing it? It's hard to know for sure.

I'm glad to hear that things are working for you though! Cheers

7

Good Afternoon!

Thanks again for your time. Regarding the other points, getting an OIDC login button on the ZAC login page, Do you have any insights?

I believe I see what might have caused some services to potentially go down when I first tried to create an edge router policy. I forgot to update my edge router with the "#public" attribute, would it make sense in that case why some services got disconnected (like the k8s api)? Or perhaps I need to review my helm values for any differences.

Thanks again.

8

I must have missed that question, do you mean like this?

image
image1232×615 146 KB

That feature has been there for a few releases now (I don't recall exactly which one). It requires the ZAC to be delivered on the same port as the OpenZiti management API.

Maybe. It is hard to know for sure. If you were connected before, adding a new router _should NOT cause a disconnect. It would be unexpected if adding a router without the attribute affected a client in that way unless the ONLY router the identity had access to was that router.

9

Regarding OIDC on the ZAC how can I confirm the port is the same?

In my helm values I have the client api set to port 1280 with the managementAPI service set to false. According to the chart comments, that should mean they share the same port.

clientApi:
  containerPort: 1280
  advertisedHost: "redacted"
  advertisedPort: 443
  service:
    enabled: true
    type: ClusterIP 
  ingress:
    enabled: true
    ingressClassName: "nginx"
    labels: {}
    annotations: {
      kubernetes.io/ingress.allow-http: "false",
      nginx.ingress.kubernetes.io/ssl-passthrough: "true",
      <redacted>
    }


managementApi:
  containerPort: 1281
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: false # enabled: true means provide this API on a separate port, otherwise share server port with clientApi
    type: ClusterIP 
  ingress:
    enabled: false
    ingressClassName: ""
    labels: {}
    annotations: {}
    tls: {}
  altIngress:
    enabled: false
    advertisedHost: ""
    ingressClassName: ""
    labels: {}
    annotations: {}
    tls: {}
  dnsNames: []
10

Yes, you understand the Ziti controller Helm chart doc correctly: unless you have set Helm values to enable the separate management API service on a separate container port, then your Ziti controller is serving the management API and console on the same port as the client API.