Hi,
Release v0.30.0 added support for running routers and controllers on a single port.
How does that work exactly? Can anyone share some configuration instructions?
I don't think it's fully documented yet but in my own testing, you just change the port for the controller's control plane and web api to the same value and it should "just work". 
Thanks!
I'm playing with exposing all the ziti services + zrok + browzer from a single 443 port with HAProxy. The plan is to use TCP passthrough for the controller and edge router, and terminate TLS in front of zac, zrok, and browzer.
I would like to use ALPN for routing the controller and edge router connections (in addition to SNI). The current ALPN protocol IDs are:
| id | purpose |
|---|---|
| ziti-ctrl | Control plane connections |
| ziti-link | Fabric link connections |
| ziti-edge | Client SDK connection to Edge Routers |
Any idea how these protocols map to the following default ports? I prefilled some values based on my guesses. Please let me know if this looks ok.
| connection | port | protocol |
|---|---|---|
| Controller control plane | 6262 | ziti-ctrl |
| Controller edge | 1280 | ziti-edge |
| Router listener | 10080 | ziti-link |
| Router edge | 3022 | ziti-edge |
| Router wss | 3023 | h2, http/1.1 (fixed thanks to @ekoby) |
Also, can all the router ports also be collapsed into a single one?
Thanks!
these look correct except for the last one:
Router WSS (port 3023) is using h2, http/1.1 (standard HTTP alpn ids) because it is using standard websocket listener
and yes, router configuration can also use a single port
1 Like
Does that also include the WSS port, or only the listener+edge ports?
it should work with all(wss+link+edge) on the same port
and let us know if you see something different
Seemed to work like a charm -- thanks!
2 Likes
I feel like I'm almost there, but there are still issues!
I have HAProxy using SNI to passthrough TCP connections directly to the controller and router containers, while TLS terminating zac/browzer/zrok.
ZAC, BrowZer seems to be working, but zrok is giving me trouble.
Zrok is not liking me anymore. I'm able to enable and environment and create a public share, but when I visit the share I get a "404" page from the zrok frontend:
ziti-zrok-frontend-1 | [2632.500] WARNING sdk-golang/ziti.(*ContextImpl).GetService: failed to get service: no apiSession, authentication attempt failed: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 404): {}
ziti-zrok-frontend-1 | [2632.500] WARNING sdk-golang/ziti.(*ContextImpl).GetService: failed to get service: no apiSession, authentication attempt failed: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 404): {}
ziti-zrok-frontend-1 | [2632.501] ERROR zrok/endpoints.GetRefreshedService: error refreshing services: failed to refresh services: no apiSession, authentication attempt failed: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 404): {}
ziti-zrok-frontend-1 | [2632.501] WARNING zrok/endpoints/publicProxy.NewHTTP.authHandler.func2: 172.18.0.3:34714 -> service 'yzsab93y4g7r' not found
ziti-zrok-frontend-1 | [2632.501] ERROR zrok/endpoints.GetRefreshedService: error refreshing services: failed to refresh services: no apiSession, authentication attempt failed: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 404): {}
ziti-zrok-frontend-1 | [2632.501] WARNING zrok/endpoints/publicProxy.NewHTTP.authHandler.func2: 172.18.0.3:34714 -> service 'yzsab93y4g7r' not found
haproxy | 77.125.98.42:51447 [25/Sep/2023:21:16:10.168] FE_https~ BE_zrok_fe/s1 0/0/1/10/11 404 31854 - - ---- 8/1/0/0/0 0/0 "GET https://yzsab93y4g7r.zrok.domain.com/ HTTP/2.0"
Also, attempting to enroll an identity on my phone now gives me a "CONTROLLER_UNAVAILABLE" error message, and the logs show:
ziti-ziti-controller-1 | [76768.826] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8440]: {error=[remote error: tls: access denied]} handshake failed
haproxy | 77.125.98.42:50980 [25/Sep/2023:21:19:28.717] FE_domain.com_tcp BE_ziti_ctrl/s1 2/0/108 4057 -- 8/8/4/4/0 0/0 req.ssl_sni:'ctrl.ziti.domain.com' req.ssl_alpn:'-'
Weird that I get the ALPN -.
Will continue debugging tomorrow... don't want to give up just yet!
The zrok issue has been fixed! Apparently the identity.json that was initially created had the previous controller URL with the old port so after manually editing the file and setting the port to 443 it now works!
@TheLumberjack/@ekoby regarding the CONTROLLER_UNAVAILABLE issue when trying to enroll an identity on the iOS app -- is it possible that it's using an outdated edge SDK version which cannot handle the single port endpoint?
Yes, I am told we are in the process of releasing ALPN support for iOS/MacOS. It's "imminent" but just isn't released yet.
SDK should handle single port without updates
Edit: make sure your JWT is generated with correct controller address. You can decode it on JWT.io.
payload iss(issuer) field should have the address of your controller. Something like this:
{
"em": "ott",
"exp": 1682269711,
"iss": "https://fd200fd3-a2d9-457f-bc0b-f9b8ee7d2898.production.netfoundry.io:443",
"jti": "339dbd56-5219-4b38-94de-6dd41f546a50",
"sub": "Nm-VK7g-3c"
}
It's non-trivial to edit the identity JSON file on iOS. Assuming the issue is that the controller port changed, your best bet is to re-enroll your iOS identity
@smilindave26 this happens when I try to enroll a new identity, and it does show me the correct controller address when I do it.
Here are the application logs from the iOS app:
(2144)[2023-09-26T16:34:20.977Z] INFO ziti-sdk:ziti_enroll.c:94 ziti_enroll() Ziti C SDK version 0.31.4 @68c3a76(HEAD) starting enrollment at (2023-09-26T16:34:20.977)
(2144)[2023-09-26T16:34:20.977Z] DEBUG ziti-sdk:jwt.c:111 load_jwt() filename is: /private/var/mobile/Containers/Shared/AppGroup/3ED4CCB6-34B0-4291-B9EB-364C66A171BB/10lFKSce6k.jwt
(2144)[2023-09-26T16:34:20.977Z] DEBUG ziti-sdk:jwt.c:77 load_jwt_file() reading JWT from file: /private/var/mobile/Containers/Shared/AppGroup/3ED4CCB6-34B0-4291-B9EB-364C66A171BB/10lFKSce6k.jwt
(2144)[2023-09-26T16:34:20.977Z] DEBUG ziti-sdk:jwt.c:104 load_jwt_file() jwt file content is:
*<edited out>*
(2144)[2023-09-26T16:34:20.977Z] DEBUG ziti-sdk:jwt.c:41 parse_jwt_content() ecfg->jwt_signing_input is:
*<edited out>*
(2144)[2023-09-26T16:34:20.977Z] INFO ziti-sdk:ziti_ctrl.c:409 ziti_ctrl_init() ctrl[ctrl.ziti.mydomain.com] ziti controller client initialized
(2144)[2023-09-26T16:34:20.977Z] VERBOSE ziti-sdk:ziti_ctrl.c:134 start_request() ctrl[ctrl.ziti.mydomain.com] starting GET[/.well-known/est/cacerts]
(2144)[2023-09-26T16:34:21.188Z] INFO ziti-sdk:ziti_enroll.c:41 verify_controller_jwt() verifying JWT signature
(2144)[2023-09-26T16:34:21.188Z] ERROR ziti-sdk:ziti_enroll.c:66 verify_controller_jwt() failed to verify JWT signature
(2144)[2023-09-26T16:34:21.188Z] ERROR ziti-sdk:ziti_ctrl.c:155 ctrl_resp_cb() ctrl[ctrl.ziti.mydomain.com] request failed: -53(software caused connection abort)
(2144)[2023-09-26T16:34:21.188Z] DEBUG ziti-sdk:ziti_enroll.c:141 well_known_certs_cb() base64_encoded_pkcs7 is: (null)
(2144)[2023-09-26T16:34:21.188Z] DEBUG ziti-sdk:ziti_enroll.c:147 well_known_certs_cb() err->message is: software caused connection abort
(2144)[2023-09-26T16:34:21.188Z] ERROR ziti-sdk:ziti_enroll.c:219 well_known_certs_cb() /Users/runner/work/ziti-sdk-swift/ziti-sdk-swift/deps/ziti-tunnel-sdk-c/build-iphoneos-arm64/_deps/ziti-sdk-c-src/library/ziti_enroll.c:148 - ZITI_JWT_VERIFICATION_FAILED => -7 (JWT verification failed)
[2023-09-26T16:34:21:189Z] ERROR CZiti:ZitiEnroller.swift:215 on_enroll() CONTROLLER_UNAVAILABLE
[2023-09-26T16:34:21:190Z] ERROR CZiti:Ziti.swift:332 enroll() Optional(Error Domain=ZitiError Code=-7 "CONTROLLER_UNAVAILABLE" UserInfo={NSLocalizedDescription=CONTROLLER_UNAVAILABLE})
[2023-09-26T16:34:22:290Z] INFO Ziti Mobile Edge:Array+ZitiIdentity.swift:25 updateIdentity() QRScan1695746056.jwt:10lFKSce6k CHANGED
I grabbed the certificate bundle that the iOS client is asking for at https://ctrl.ziti.mydomain.com/.well-known/est/cacerts and it contains the correct certificate chains, which include my wildcard cert for *.ziti.mydomain.com.
How I did it:
$ curl https://ctrl.ziti.mydomain.com/.well-known/est/cacerts | base64 -d > certs.p7b
$ openssl pkcs7 -inform DER -in certs.p7b -text -print_certs
Any ideas?
before SDK can fetch pkcs7 bundle server has to be verified. SDK is using controller cert to verify it was used to signed the JWT.
Can you get controller cert and JWT and validate them on JWT.io?
I grabbed the first certificate using: openssl s_client -showcerts -servername ctrl.ziti.mydomain.com -connect ctrl.ziti.mydomain.com:443. I also grabbed the private key (which is set as the server_key under alt_server_certs in the controller identity).
I then pasted the JWT I was trying to enroll and it passed validation :/.
Payload looks like this:
{
"iss": "https://ctrl.ziti.mydomain.com:443",
"sub": "10lFKSce6k",
"aud": [
""
],
"exp": 1696278247,
"nbf": 1695673447,
"jti": "d565e59b-6434-4e26-9e6c-c1383e40a52d",
"em": "ott"
}
The FIRST issue...
I've been digging through this on my side trying to replicate your issue. I think I have reproduced your problem on my side. I suspect you followed the browzer example guide, which works fine for browzer but then, I suspect you changed the 'address' of the controller so the jwts all use the "non-self-signed CA"... This gets complex, hopefully this will make sense.
When configuring the controller with alt_server_certs, they are used for exactly what it sounds like, as "server" certs. That's fine for accessing the API or connecting to a 'server' port.... however, it's not fine if you want to change the advertised address identities use/see (as it appears you may have shown in your jwt above)... IF you change the address in the controller's configuration, you must change the identity server_cert and key section of your edge api due to how OpenZiti's clients verify the authenticity of enrollment tokens (jwts)...
When an enrollment token is delivered to a client, the jwt is signed by the controller (server). The key used to sign the jwt is the one specified in the identity section (not the alt_server_cert section). When the client tries to enroll, it will read the issuer field, get the server cert from the specified issuer and verify the jwt was signed by that issuer... The problem is that if you configure the external pki in alt_server_certs, the key will not match the server certificate presented and the enrollment will fail. That's what I believe is the first problem you have. Please confirm the identity section of your setup.
An Example:
As an example, i have stood up: https://ctrl.clint.demo.openziti.org:8441 from the LetsEncrypt PKI. My identity section first looked like this:
ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-root-ca/certs/ip-172-31-11-231-edge-controller-root-ca.cert"
key: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/keys/ec2-13-58-222-94.us-east-2.compute.amazonaws.com-server.key"
server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/certs/ec2-13-58-222-94.us-east-2.compute.amazonaws.com-server.chain.pem"
cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/certs/ec2-13-58-222-94.us-east-2.compute.amazonaws.com-client.cert"
alt_server_certs:
- server_cert: "/etc/letsencrypt/live/clint.demo.openziti.org/fullchain.pem"
server_key: "/etc/letsencrypt/live/clint.demo.openziti.org/privkey.pem"
Looks fine, self-signed CA in the identity section, LetsEncrypt in the alt_server_certs... But, when I changed my address field to match from ec2-13-58-222-94.us-east-2.compute.amazonaws.com:8441 to: ctrl.clint.demo.openziti.org:8441, I could no longer enroll a single identity anywhere.
I had to update my identity section to look like this for it to work properly, which I think is "unintuitive" at best:
identity:
ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-root-ca/certs/ip-172-31-11-231-edge-controller-root-ca.cert"
key: "/etc/letsencrypt/live/clint.demo.openziti.org/privkey.pem"
server_cert: "/etc/letsencrypt/live/clint.demo.openziti.org/fullchain.pem"
cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/certs/ec2-13-58-222-94.us-east-2.compute.amazonaws.com-client.cert"
alt_server_certs:
- server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/certs/ec2-13-58-222-94.us-east-2.compute.amazonaws.com-server.chain.pem"
server_key: "/home/ubuntu/.ziti/quickstart/ip-172-31-11-231/pki/ip-172-31-11-231-edge-controller-intermediate/keys/ec2-13-58-222-94.us-east-2.compute.amazonaws.com-server.key"
Notice how I have moved the server/key and alt server/key. Also note that the 'cert' in this situation is just not used at this time. That cert and key won't match. I don't know if this will be a problem in the future, but I'll raise it as an issue to the team....
The Second Issue...
The second issue you'll run into is that the currently deployed Windows and MacOS clients are not able to use LetsEncrypt certificates at this time. This problem is fixed in the latest unreleased versions, due to be released soon.... (I don't know when) I don't know if you're using LetsEncrypt or some other CA, but that will be the next problem you face from a MacOS perspective.
We need to tighten up the doc and experience around alt certs for sure. This wasn't obvious nor easy for me to figure out either... We'll work on getting that done too.
I hope this helps.
First of all, I really appreciate your time with troubleshooting this issue!
Second, you're right on both counts - I did rename the controller address after initial deployment and I did add alt_server_certs when I added Let's Encrypt certs.
Swapping the alt_server_certs with the ones in identity didn't seem to help though, but that is possibly because my proxy config - I will try to go back to using different ports and see if it makes any difference. There's also another identity section at the root of the YAML and a signingCert key under edge/api/enrollment and I'm not sure what they are used for exactly. I wonder what else could be silently broken because of this and how can I make sure everything is correctly configured cert-wise.
Note that trying to deploy via iOS gives this controller log msg
ziti-ziti-controller-1 | [77525.262] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8440]: {error=[remote error: tls: access denied]} handshake failed
I also tried to enroll an identity on the Windows client, and it actually was able to proceed further by actually able to download the certs:
[2023-09-26T21:22:57.509Z] INFO ziti-sdk:ziti_enroll.c:92 ziti_enroll() Ziti C SDK version 0.33.2 @e06b76c(HEAD) starting enrollment at (2023-09-26T21:22:57.509)
[2023-09-26T21:22:57.509Z] VERBOSE ziti-sdk:jwt.c:124 load_jwt_content() jwt file content is:
eyJhbGciOiJFUzI1NiIsImtpZCI6IjUwMDMyMGMwZmEyNTY0ZDcwNTkxMDE4NTFmZWY1ODY1MTllYmUwZWEiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL2N0cmwueml0aS52ZWxiYXVtLmNvbTo0NDMiLCJzdWIiOiJXb1ZMYWJKVnYiLCJhdWQiOlsiIl0sImV4cCI6MTY5NjM2ODE2NCwibmJmIjoxNjk1NzYzMzY0LCJqdGkiOiI1YWY4MjcyNi1kNjUwLTQ4ZmItYmZmMi01ZTczZmVhMWNiMzciLCJlbSI6Im90dCJ9.xU4Etve9f3WZ2aEQKN0kGpFeJl3LE94oefy5wOdbZ7XbyZMC1uqvyYX2NGdls662bshl609cLn4OO5z52FM_6g
[2023-09-26T21:22:57.509Z] DEBUG ziti-sdk:jwt.c:36 parse_jwt_content() ecfg->jwt_signing_input is:
eyJhbGciOiJFUzI1NiIsImtpZCI6IjUwMDMyMGMwZmEyNTY0ZDcwNTkxMDE4NTFmZWY1ODY1MTllYmUwZWEiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL2N0cmwueml0aS52ZWxiYXVtLmNvbTo0NDMiLCJzdWIiOiJXb1ZMYWJKVnYiLCJhdWQiOlsiIl0sImV4cCI6MTY5NjM2ODE2NCwibmJmIjoxNjk1NzYzMzY0LCJqdGkiOiI1YWY4MjcyNi1kNjUwLTQ4ZmItYmZmMi01ZTczZmVhMWNiMzciLCJlbSI6Im90dCJ9
[2023-09-26T21:22:57.510Z] DEBUG ziti-sdk:ziti_ctrl.c:408 ziti_ctrl_init() ctrl[ctrl.ziti.mydomain.com] ziti controller client initialized
[2023-09-26T21:22:57.510Z] VERBOSE ziti-sdk:ziti_ctrl.c:133 start_request() ctrl[ctrl.ziti.mydomain.com] starting GET[/.well-known/est/cacerts]
[2023-09-26T21:22:57.510Z] TRACE ziti-edge-tunnel:tun.c:303 tun_read() starting read
[2023-09-26T21:22:57.663Z] DEBUG ziti-sdk:ziti_enroll.c:39 verify_controller_jwt() verifying JWT signature
[2023-09-26T21:22:57.663Z] DEBUG ziti-sdk:ziti_enroll.c:67 verify_controller_jwt() JWT verification succeeded!
[2023-09-26T21:22:57.739Z] VERBOSE ziti-sdk:ziti_ctrl.c:168 ctrl_resp_cb() ctrl[ctrl.ziti.mydomain.com] received headers GET[/.well-known/est/cacerts]
[2023-09-26T21:22:57.739Z] VERBOSE ziti-sdk:ziti_enroll.c:144 well_known_certs_cb() base64_encoded_pkcs7 is: *<snipped>*
[2023-09-26T21:22:57.741Z] DEBUG ziti-sdk:ziti_enroll.c:157 well_known_certs_cb() CA PEM len = 15764
[2023-09-26T21:22:57.741Z] TRACE ziti-sdk:ziti_enroll.c:158 well_known_certs_cb() CA PEM:
-----BEGIN CERTIFICATE-----
*<snipped>*
[2023-09-26T21:22:57.745Z] DEBUG ziti-sdk:ziti_ctrl.c:408 ziti_ctrl_init() ctrl[ctrl.ziti.mydomain.com] ziti controller client initialized
[2023-09-26T21:22:57.745Z] VERBOSE ziti-sdk:ziti_ctrl.c:133 start_request() ctrl[ctrl.ziti.mydomain.com] starting POST[/enroll?method=ott&token=5af82726-d650-48fb-bff2-5e73fea1cb37]
[2023-09-26T21:22:57.897Z] ERROR ziti-sdk:ziti_ctrl.c:154 ctrl_resp_cb() ctrl[ctrl.ziti.mydomain.com] request failed: -4079(software caused connection abort)
[2023-09-26T21:22:57.897Z] ERROR ziti-sdk:ziti_enroll.c:235 enroll_cb() failed to enroll with controller: https://ctrl.ziti.mydomain.com:443 CONTROLLER_UNAVAILABLE (software caused connection abort)
[2023-09-26T21:22:57.897Z] ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:319 tunnel_enroll_cb() enrollment failed: CONTROLLER_UNAVAILABLE(-3)
[2023-09-26T21:22:57.897Z] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:227 on_command_resp() resp[0,len=56] = {"Success":false,"Error":"enrollment failed","Code":500}
The controller reported an error which is a bit different than the one reported when enrolling on iOS:
ziti-ziti-controller-1 | [76977.945] ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:8440]: {error=[local error: tls: bad record MAC]} handshake failed
This section is your "control plane" section. That's relevant to how routers talk to routers, routers to controllers etc. I wouldn't think that would matter. I think maybe this week's Ziti TV can be dedicated towards a 'working session' where I basically setup a network with a couple of routers, external certs, and make sure data passes from identity to identity properly.
What does it mean to 'deploy via iOS'? You're just trying to enroll an identity, right? I'm very sure the MacOS (I dont' have an iphone but it's the same code, generally) and the windows clients won't work with LetsEncrypt certs until this next update comes out for them. You could try the latest build of "raw" ziti-edge-tunnel though: Release v0.22.7 · openziti/ziti-tunnel-sdk-c · GitHub that should work on MacOS/Windows but it's not the nice, native UI app, it's just a command line tool.
This section is your "control plane" section. That's relevant to how routers talk to routers, routers to controllers etc. I wouldn't think that would matter.
I see, so I guess for that section the alt_server_certs should be good enough.
I think maybe this week's Ziti TV can be dedicated towards a 'working session' where I basically setup a network with a couple of routers, external certs, and make sure data passes from identity to identity properly.
That would be cool indeed!
What does it mean to 'deploy via iOS'? You're just trying to enroll an identity, right?
Oh yeah, sorry, I meant to enroll (it's 1am here so please forgive the typos :)).
I forgot to mention that I used a pre-release of the Windows client: Release 2.1.17 · openziti/desktop-edge-win · GitHub -- not sure if this version is supposed to have the Let's Encrypt cert support or not.
I can send you a DM with a 2.1.18 build that will work (it's the one I'm using locally, to duration test) just know that if we release a 'real' build, you'll want to either uninstall it and install the 'released' version to pick up any other updates that might be in it... I'll shoot you a link to it you can try
1 Like