Unable to serve ZAC over BrowZer

BrowZer
1

I'm sorry if I'm exaggerating the questions here, but I'm currently trying out a few things.

I'm trying to serve the console with browZer.
I've set it up using letsencrypt certificates and the docker image from Package ziti-browzer-bootstrapper · GitHub which is 0.63.4.
Identity provider is Microsoft Entra ID.

It seems to be working in general, as another site is being served (even if this error occurs here)

I've then setup a second vhost for the console, same idp_client_id.
Authentication with the identity provider succeeds.

But I then run in the following error:

BrowZer Runtime code: 1017

Cannot Reach Ziti Controller [https://ctrl.ziti.xyz.de:1280/edge/client/v1]

If I type the URL in the browser, the page is served perfectly fine.

I see the following error in the browser console

ziti-browzer-runtime-856546f8.js:169341 Refused to connect to 'https://ctrl.ziti.xyz.de:1280/edge/client/v1/version' because it violates the following Content Security Policy directive: "connect-src 'self' www.googletagmanager.com www.google-analytics.com openstreetmap.org ws: wss: https://login.microsoftonline.com/d313f676-90ff-4c91-a3b0-0507183e0b00/v2.0 https://*.netfoundry.io:* https://*.cloudziti.io wss://*.netfoundry.io:* data:".
ziti-browzer-runtime-856546f8.js:169341 Refused to connect to 'https://ctrl.ziti.xyz.de:1280/edge/client/v1/version' because it violates the document's Content Security Policy.
ziti-browzer-runtime-856546f8.js:169341 Refused to connect to 'https://ctrl.ziti.xyz.de:1280/edge/client/v1/authenticate?method=ext-jwt' because it violates the following Content Security Policy directive: "connect-src 'self' www.googletagmanager.com www.google-analytics.com openstreetmap.org ws: wss: https://login.microsoftonline.com/d313f676-90ff-4c91-a3b0-0507183e0b00/v2.0 https://*.netfoundry.io:* https://*.cloudziti.io wss://*.netfoundry.io:* data:".
ziti-browzer-runtime-856546f8.js:169341 Refused to connect to 'https://ctrl.ziti.xyz.de:1280/edge/client/v1/authenticate?method=ext-jwt' because it violates the document's Content Security Policy.

BrowZer log

root@ad-ztna01:~# docker logs -f -n0 ziti-browser-ziti-http-agent-1 
{"error":"Cannot Reach Ziti Controller [https://ctrl.ziti.xyz.de:1280/edge/client/v1]","error_code":1017,"level":"error","message":"Possible configuration | certificates issue exists.","timestamp":"2024-08-06T13:51:24.136Z","version":"0.63.4"}

Router log

Aug 06 13:52:43 ad-ztna01 ziti[113426]: {"circuitCount":7,"file":"github.com/openziti/ziti/controller/handler_ctrl/circuit_confirmation.go:47","func":"github.com/openziti/ziti/controller/handler_ctrl.(*circuitConfirmationHandler).HandleReceive","level":"info","msg":"received circuit confirmation request","routerId":"pHbNlOUrfj","time":"2024-08-06T13:52:43.155Z"}

Looks to me that the browser is blocking something because of some content policy.
Any idea?

2

@pgross I think I see the problem. Please stand by while I attempt to reproduce it.

3

@pgross I believe I have corrected the issue you reported.

I have a branch-build available. When you have a moment, please give ghcr.io/openziti/ziti-browzer-bootstrapper:pr310.751 a try, and let me know if things have improved for you.

If things look good, I'll do the release.

4

@curt cool, that seems to be working.
Just tested it and after clearing site data, all worked.

Thank you!

5

Terrific. I'll do the release now.

1 Like
6

0.64.5 has been released

1 Like
7

I'm sorry that I have to reopen this.

I've tested it thoughly today.
It is sometimes working in incognito mode of the browser. I have the feeling that this depends on what certificate I choose to be presented by the browser. The beforementioned error in the console are gone. So this might be a different problem.

In the normal non-incognito mode I don't get the chance to select a certificate.

The controller log shows the following:

Aug 07 10:34:57 ad-ztna01 ziti[962]: {"error":"token is unverifiable: error while executing keyfunc: key for kid KQ2tAcrE7lBaVVGBmc5FobgdJo4, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:845","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","time":"2024-08-07T10:34:57.541Z"}
Aug 07 10:34:58 ad-ztna01 ziti[962]: {"error":"token is unverifiable: error while executing keyfunc: key for kid KQ2tAcrE7lBaVVGBmc5FobgdJo4, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:845","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","time":"2024-08-07T10:34:58.537Z"}
Aug 07 10:34:59 ad-ztna01 ziti[962]: {"error":"token is unverifiable: error while executing keyfunc: key for kid KQ2tAcrE7lBaVVGBmc5FobgdJo4, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:845","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","time":"2024-08-07T10:34:59.578Z"}
Aug 07 10:35:00 ad-ztna01 ziti[962]: {"error":"token is unverifiable: error while executing keyfunc: key for kid KQ2tAcrE7lBaVVGBmc5FobgdJo4, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:845","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","time":"2024-08-07T10:35:00.082Z"}
Aug 07 10:35:00 ad-ztna01 ziti[962]: {"file":"github.com/openziti/ziti/controller/network/fault.go:32","func":"github.com/openziti/ziti/controller/network.(*Network).fault","level":"info","msg":"network fault processing for [2] circuits","time":"2024-08-07T10:35:00.904Z"}
Aug 07 10:35:02 ad-ztna01 ziti[962]: {"error":"token is unverifiable: error while executing keyfunc: key for kid KQ2tAcrE7lBaVVGBmc5FobgdJo4, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:845","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","time":"2024-08-07T10:35:02.399Z"}
Aug 07 10:35:02 ad-ztna01 ziti[962]: {"error":"token is unverifiable: error while executing keyfunc: key for kid KQ2tAcrE7lBaVVGBmc5FobgdJo4, not found","file":"github.com/openziti/ziti/controller/env/appenv.go:845","func":"github.com/openziti/ziti/controller/env.(*AppEnv).getJwtTokenFromRequest","level":"error","msg":"error during JWT parsing during API request","time":"2024-08-07T10:35:02.404Z"}

Aug 07 10:39:39 ad-ztna01 ziti[845]: {"_context":"classic/wss:0.0.0.0:8505","file":"github.com/openziti/transport/v2@v2.0.138/wss/listener.go:60","func":"github.com/openziti/transport/v2/wss.(*wssListener).handleWebsocket","level":"info","msg":"entered","time":"2024-08-07T10:39:39.018Z"}
Aug 07 10:39:45 ad-ztna01 ziti[845]: {"circuitId":"wQ6YVF3Lv","ctrlId":"NetFoundry Inc. Client P-RbbQIiR","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":1037224000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-08-07T10:39:45.575Z"}
Aug 07 10:39:45 ad-ztna01 ziti[845]: {"circuitCount":1,"ctrlId":"NetFoundry Inc. Client P-RbbQIiR","file":"github.com/openziti/ziti/router/forwarder/scanner.go:100","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","level":"warning","msg":"sent confirmation for circuits","time":"2024-08-07T10:39:45.580Z"}
Aug 07 10:39:45 ad-ztna01 ziti[845]: {"circuitCount":1,"ctrlId":"NetFoundry Inc. Client P-RbbQIiR","file":"github.com/openziti/ziti/router/forwarder/faulter.go:117","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run.func2","level":"warning","msg":"reported forwarding faults","time":"2024-08-07T10:39:45.916Z"}

If I open the website using the windows tunneler, everything works.

8

Hi @pgross

The problem you are experiencing can sometimes appear when MSFT Entra is used as the IdP. The key for kid ..., not found error can sometimes manifest under the following circumstances:

  • you create your external-jwt-signer
  • the Ziti Controller hits the IdP's JWKS endpoint to learn about the IdP's various kid's
  • the IdP (MSFT Entra) provides its list of kid's
  • ...time passes
  • The IdP spawns additional/new kid's (for reasons currently unknown to me)
  • user starts a fresh browZer session, authenticates with the IdP, and receives a JWT, and the JWT contains one of the "additional/new" kid values (that the Ziti Controller is unaware of)
  • the above JWT is used by the Ziti BrowZer Runtime (ZBR) to authenticate with the Controller
  • Controller rejects the auth attempt due to unrecognized kid in the JWT

The controller will hit the JWKS endpoint of all configured external JWT signers (a.k.a. IdP's) when the Controller starts up... so, the first thing you can try is to simply stop/restart your Controller, then retry your browZer auth flow again, and you should be fine.

SIDE NOTE:
Please hit the /version endpoint of your Controller, then post the response here. I'd like to know which version of Ziti you are running. I ask because some updates to the above-mentioned flow were shipped not too long ago that were intended to automatically learn about newly appearing kid values in the IdP. Either you do not have those updates, or, those updates are not working in your environment.

9

There was a bug prior to OpenZiti v1.1.2 (changelog: ziti/CHANGELOG.md at main · openziti/ziti · GitHub, issue #2002). If you are on a version older than that, then it would most likely be that bug. If you are on a version newer than that then we have a different issue on our hands.

10

Hey there, I’m running version 1.1.7

buildDate":"2024-07-16T13:15:20Z","capabilities":[],"revision":"94013fe4af89","runtimeVersion":"go1.22.5","version":"v1.1.7"
11

OK, thanks, Pascal.

Can you send us the entire Controller log for the last couple of days?
Can you describe for us what (if any) config changes you made in your Entra account that were done after you initially set up the external-jwt-signer in your Ziti network?

Also, let us know if bouncing the Controller got you past the "kid not found" error.

12

Restarting the controller doesn't seem to bring any change.

On the first try I got asked about a certificate - I chose randomly.

image
image1869×874 48.3 KB

Furthermore, the URL changes to browzer_error when I don't choose fast enough but I don't see any error page.

13

here is a link to the log file

14

Hi Pascal, can you show us the JWKS url for your ext-jwt-signer? You can list it via the CLI with the following cmd: ziti edge list ext-jwt-signers

15

Is it safe to post it here? I'd rather send it via pm

16

By the way, the other site I have published over BrowZer (the Technische Baubestimmungen I shared with you), is loading (without the style sheets ofc).

Another point that could be interesting: I'm reusing the idp_client_id - hope that is valid?

17

Got the PM regarding JWKS. Thanks!

18

The kid that the Controller says is "not found" is definitely in the JWKS keys list. So, if a stop/restart of your Controller doesn't pick it up, we will need @andrew.martinez to weigh in here.

19

@pgross The lack of style sheets with the Technische Baubestimmungen web app is understood. I have a fix that handles the @import within inline style tags, but I haven't released it yet. That fix is intermingled with other updates I have that are aimed at supporting the synchronous XHR used by the Dojo framework employed by Technische Baubestimmungen, so it might be a little longer before I present a branch-build to you.

20

That sounds great, thank you for your effort!