BrowZer with Zitadel: "Check that this 'externalId' exists and has case-sensitive match"

BrowZer
1

Hi all,

with much difficulties, I was able to install the quickstart OpenZiti setup and also the BrowZer bootstrapper.
I am using Zitadel as my IdP, but spun up a Keycloak instance also to double-check if the errors I got were IdP or Ziti sourced.

Currently, I am struggling with the login to the example brozac app, so the Ziti Admin via BrowZer.
I got as far as being able to log in via the paired IdP, but now I get the following error:

BrowZer Runtime code: 1001

User [abc@example.com] cannot be authenticated onto Ziti Network

Check that this 'externalId' exists and has case-sensitive match

During the setup, I created the relevant identity in Ziti and everything necessary around it.
However, BrowZer seems to pull the wrong OpenID value from the IdP token.
I specified "preferred_username" (essentially the full login name to uniquely identify a Zitadel user across all realms) as the wanted claims property in the connected jwt signer, but it doesnt work.

How can I check why the wrong claim is being used?

2

Hi @etec-masterofsynapse, welcome to the community and BrowZer! (and OpenZiti/zrok)

This part of the process is often finicky to put it kindly. I'm not sure how to check it with Zitadel, but with Keycloak there's a nice way with the "hard to find" (imo) UI. Go to clients->your client you're using. Click on "Client scopes" and finhd the "evaluate" tab that is grey and really hard to notice...

image
image788×394 37.1 KB

from there you can pick your user then choose your access or ID token and inspect it

image
image1460×698 67.2 KB

Right now, browzer is using an ID token, but that will likely be changing soon, just an fyi.

I'm not sure how to do it in Zitadel, but there must be some sort of equivalent functionality in there. Otherwise, you can log in and try to capture the token from the browser interactions.

Look for a 401 from the "authenticate" url and scrape out the "Bearer" token (without the bearer) and use jwt.io to view the token.

image
image937×725 56.6 KB

Hope that helps! Nice job so far! :slight_smile:

3

Hey,

thanks for the helpful input!

Because I also dont know off the top of my head how to internally check the info of a token, I went the jwt.io way and saw there, that the wanted preferred_username field is missing fully.
Even stranger, a whole lot of fields are also missing that are usually there.

Can I request additional scopes from the jwt signer in Ziti? Usually, in OpenID configs, there is a field to specify openid, email, profile and the like.

4

@curt would probably have to comment for sure (or I'd have to go paw through the code), but I don't believe there's any way to request additional scopes with BrowZer at this time. Other OpenZiti tech allows you to specifiy scopes, but I don't think it's integrated into BrowZer -- but I might be wrong. I don't know all there is to know about BrowZer.

How did you get the token to look at? If it's from the latter method I showed, where you look at the /authenticate url, it might be because the ID token returned doesn't contain the fields you expect to see. The fields might be in the access token. For now, BrowZer is using an ID token, just keep that in mind.

I would expect there's some way to add a claim to requests in some way with Zitadel. Maybe Configuring Custom Claims in ZITADEL ? You might have to force it onto authentication requests for the timebeing?

5

Well, according to the first table on this page Claims in ZITADEL | ZITADEL Docs preferred_username is always contained in the ID token, and never in the access token.

I tried to program the right javascript to force output that scope too, via Actions, but since I am not well-versed in that, I seem to fail.
I think I got it right, but once I enabled the relevant action, BrowZer now shows me the runtime error 1013, where the token isnt recognized as an SPA.
Speaking about that, I currently also have the same problem with Keycloak.
I followed guides online to ensure that I have PKCE enabled on the client, but there is still something missing it seems.

And on the topic of Ziti vs BrowZer OpenID scopes:
Isnt BrowZer just consuming the token Ziti requests from the jwt signer?
At least thats how it looks to me who cant know, since the jwt signer and auth policies are kept in the Ziti Admin Center.

6

I will give it a try. Are you self-hosting zitadel or are you using their SaaS offering? I'll start off with their self-hosted offering and hope that it's close enough to the SaaS if you're not self-hosting it...

7

I am using the self-hosted variant with Docker Compose, since I want to rebuild all Zero Trust components within my own DCs and for my MSP customers.
Thats also how I found OpenZiti as a replacement for Cloudflare Access and Cloudflare Tunnel.

8

Cool. I've not done this for a minute so, gimme a bit and I'll see what I can come up with. If you're interested to see how I configure keycloak you can look at the "browzer.configure.keycloak.sh" script I wrote for myself a while back openziti-scripts/ziti-zrok-browzer/browzer.configure.keycloak.sh at main · dovholuknf/openziti-scripts · GitHub. the repo has a whole bunch of bash that I use to setup my own environment but does a whole bunch more too... You probably don't need all that for now, but figured I'd share in case you were interested.

I'll get my zitadel up and running and try it out and report back when i can...

1 Like
9

I'm stumped with zitadel initialization for an external domain. I've been reading their doc and trying all kinds of things but didn't have any luck yet. I'll probably keep trying tomorrow. I reached out to their forum but i'm waiting for a response. From what I can tell the instance is deployed adequately, I just can't authenticate to the console yet. Maybe you have a pointer on how to do that initial setup for zitadel? Other than that, I'll keep trying as I have time.

I used their docker compose base setup, I have TLS enabled and it's online (you can access it at https://zitadel.clint.demo.openziti.org:8558/).

EDIT: i ended up getting through my login issue. I'll see if I can get zitadel working.

10

Good to see via the edit that you got it working.

I have to say, even though Zitadel is a nice app and much tighter and most times easier than Keycloak, both Zitadel and Keycloak have massive holes in the documentation, making it very hard for a newcomer, for example me a few weeks back, to deploy them successfully.

I know I also struggled with the external domain implementation.
My latest step to get those to work was to just deploy a regular reverse proxy with Nginx, definitely enabling external SSL in the Docker Compose, and then it got around and started working.

11

Yeah. It's easy enough once you get TLS "working" but finding the env vars to put in the right places was more difficult than it should have been. I footgunn'ed myself by using wildcard dns. Zitadel is deployed at zitadel.clint.demo.openziti.org, so when you login, you need to duplicate the zitadel bit... :confused: making the login name:

zitadel-admin@zitadel.zitadel.clint.demo.openziti.org

go figure...

Anyway, after getting through that travesty, it was pretty easy for me to configure my browzer instance for zitadel using federated to GitHub/Google auth. The email was in the ID token just fine and things worked fine.

The magic for me was these envirnonment variables declared:

      - 'ZITADEL_TLS_CERTPATH=/etc/certs/selfsigned.crt'
      - 'ZITADEL_TLS_KEYPATH=/etc/certs/selfsigned.key'

and these volumes:

      - '/path/to/lets/encrypt/fullchain.pem:/etc/certs/selfsigned.crt'
      - '/path/to/lets/encrypt/privkey.pem:/etc/certs/selfsigned.key'

Going back to your issue now that I have a workable demo system, i think you're stuck with the ID token not being valid? I didn't need to add an action/trigger for it to work for me.

Remind me where you're at?

12

Right, the email is contained, thats why its finding A user, but the wrong one in my Ziti instance.

Zitadel can use multiple things as loginname.
Since I have a B2B Zitadel instance, with tenants for apps and tenants for customers, I need to be able to have multiple users with the same mail address, so I am using the preferred_username field to login the user, and that string was also used to create the identity in Ziti.
So, in your example, with the zitadel-admin, the identity in Ziti is

zitadel-admin@zitadel.zitadel.clint.demo.openziti.org

Meaning, the preferred_username field must be in the ID token, and thats whats currently not working.

13

Did you toggle the "User Info inside ID token"?

image
image976×515 17.4 KB

After I did that, the ID token has my preferred_username:

image

14

I did. However, you selected Auth Token Type as Bearer, I currently have JWT.
Seems like that makes a difference.

15

I couldn't remember if it did (i tried a bunch of things) which is why i left it in that screen cap. :slight_smile:

Sounds like you're moving along again though - lemme know if you need anything else!

16

If helpful, I made my ext-jwt-signer and auth-policy via cli (forgive my mixed casing):

ziti_object_prefix="zitadel"
issuer="jwt issuer field here"
jwks="zitadel url/oauth/v2/keys"
ZITI_BROWZER_CLIENT_ID="your-client-id"

ext_jwt_signer=$(ziti edge create ext-jwt-signer "${ziti_object_prefix}-ext-jwt-signer" "${issuer}" --jwks-endpoint "${jwks}" --audience "${ZITI_BROWZER_CLIENT_ID}" --claims-property email)
echo "ext jwt signer id: $ext_jwt_signer"

auth_policy=$(ziti edge create auth-policy ${ziti_object_prefix}-auth-policy --primary-ext-jwt-allowed --primary-ext-jwt-allowed-signers ${ext_jwt_signer} --secondary-req-ext-jwt-signer ${ext_jwt_signer})
echo "auth policy id: $auth_policy"

EDIT: you'll want to change that claims property too, i noticed i used email

17

I changed the Token Type, and also recreated the app in Zitadel, to make sure that no bug is stopping me.
But, both did nothing, the preferred_username scope is still not in the ID token.

Regarding signer and auth creation, I followed the quick start guide, and I did it via CLI as well.

So, for now I will try to get Keycloak to work... :frowning:

18

I tried again with Keycloak, but had no luck there also.
I followed both this guide Keycloak for OpenZiti BrowZer and the scripts you linked me, but my Keycloak config is essentially the same.

But I still get the error that BrowZer thinks the token isnt a SPA.

I honestly dont understand what I did so wrong in my deployments that both Zitadel and Keycloak dont work...

19

This is exactly how I configured Zitadel a third time (once last night, twice just now). Each time it works for me :frowning: :

image
image1313×640 28 KB

After that, I used the client id it generates: 290310781285761027 and inspect a token. It comes back without the preferred_username.

I then go to "Token Settings" and toggle this on and try again:

image

At that point I get preferred_username returned.

I JUST installed it last night. It's v2.63.4. Maybe it's a bug/feature that was fixed in a later version?

Can you show me the action browzer page? If you look at the browzer logs, are there any errors that spit out? Does the openziti controller have any useful/relevant log in it?

20

One relevant difference I see is that your Zitadel is running on the same host as your Ziti instances.
Mine runs on a different machine, with a reverse proxy in between etc.
And, both in your scripts and also in the first screenshot just now, I notice you added the localhost... redirect URI. I dont do that, especially because the ports between Ziti and Zitadel arent open.

I also thought that its a problem with my version, I updated to 2.63.4, didnt change anything.

What action browzer page do you mean?

Regarding logs, I didnt find any special log in the quickstart folder or in /var/log, so I looked in syslog and found some:

2024-10-21T15:23:59.426425+00:00 epszititest ziti[732]: {"_context":"tls:0.0.0.0:8441","error":"EOF","file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"xx:1371","time":"2024-10-21T15:23:59.425Z"}

2024-10-21T15:24:00.936035+00:00 epszititest ziti[732]: {"error":"token is unverifiable: error while executing keyfunc: key for kid xx, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:862","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","tim>

2024-10-21T15:24:01.698891+00:00 epszititest ziti[732]: {"authMethod":"ext-jwt","file":"github.com/openziti/ziti/controller/model/authenticator_mod_ext_jwt.go:394","func":"github.com/openziti/ziti/controller/model.(*AuthModuleExtJwt).process","level":"error","msg":"encountered 1 candidate JWTs and all failed to validate for primary authentication, see the following log messages>

2024-10-21T15:24:01.699076+00:00 epszititest ziti[732]: {"audience":"xx","authMethod":"ext-jwt","error":"audience validation failed: audience value is invalid, no audiences matched the expected audience","extJwtSignerId":"xx","file":"github.com/openziti/ziti/controller/model/authenticator_mod_ext_jwt.go:86","func":"github.com/openziti/ziti/co>

2024-10-21T15:24:01.718264+00:00 epszititest ziti[732]: {"error":"token is unverifiable: error while executing keyfunc: key for kid xx, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:862","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","tim>

2024-10-21T15:24:01.746159+00:00 epszititest node[1315]: {"error":"User [xx] cannot be authenticated onto Ziti Network","error_code":1001,"level":"error","message":"Check that this 'externalId' exists and has case-sensitive match","timestamp":"2024-10-21T15:24:01.744Z","version":"0.74.0"}