Jumping into this thread for a second... Did you perhaps remove the "control plane CA"? aka the "ziti controller" CA from the bundle? the quickstarts were always created in a complex fashion, to demonstrate what is possible. As such - they ship with three different root CAs. One for the control plane, one for the edge apis and one for identities.
Between 0.26.8 and now, changes to the SDKs have changed the way the PKI that was generated works. Now, you must ensure the ca bundle contains root CAs only, and the full chain is used for the server certificates presented to complete the chain of trust.
It's possible the root ca for the control plane (router to router, router to controller, edge client to router) was removed. It's also possible the older router does not use the full chain.
You can test that your bundle is properly setup by doing two steps...
- download the bundle and unwrap it
- use the bundle with openssl to verify connectivity
In one copy/pastable block that looks like this (change ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT of course):
ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT=ec2-3-18-113-172.us-east-2.compute.amazonaws.com:8441
curl -sk https://"${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT}/.well-known/est/cacerts" \
| openssl base64 -d \
| openssl pkcs7 -inform DER -outform PEM -print_certs -text -out cacerts.pem
openssl s_client \
-connect "${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT}" \
-CAfile cacerts.pem < /dev/null \
| grep Verify
You need to make sure you see: Verify return code: 0 (ok). Assuming you do, I would expect everything to work.
Can we check your router's cert it's presenting to see if it's returning a chain? Dunno if @ekoby has anything else to add.
1 Like
Thank you Clint, I'll try that. I just yesterday followed your PKI Ziti TV
I'm still not 100% sure I understand correctly how the chain verification works with Ziti.
-
For the CA bundle, certificate order doesn't matter, correct? It's not directly a chain file but rather the CAs are searched within that file, right?
-
When the CA bundle only contains the Root Certs, where exactly on the server side are intermediary certs stored? Do I have to supply a chain with the leaf cert AND intermediary certs by every identity?
correct..
Assuming the quickstart was used and the locations haven't changed, all the PKI (keys, intermediates, roots, etc) will ALL be located at $HOME/.ziti/quickstart/$(hostname)/pki. Technically, you should open the controller config file and inspect the location, but it's 99.9% that they're there...
The identities need to have the CA bundle (which I expect they'll have) and the servers they connect to need to present the full chain, so identities would only need CAs in that situation. It should be find if the identities have the same certs as the servers present, hopefully that makes sense
1 Like
That works:
openssl s_client \
-connect "${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT}" \
-CAfile modified_cas.pem < /dev/null \
| grep Verify
Connecting to 46.38.242.110
depth=2 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=zt.mycompany.de-root-ca Root CA
verify return:1
depth=1 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=zt.mycompany.de-intermediate
verify return:1
depth=0 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=zt.mycompany.de server certificate
verify return:1
DONE
Verify return code: 0 (ok)
But just for the sake of completeness I want to add that I didn't pull the cacerts from the controller but just composed it locally, since on the controller it's still the "old" CAs with the intermediary in it.
Do I need to update this file right now as well for the testing or is its purpose just to supply newly generated identities with the CAs?
Gotcha.. If you update the file the controller is configured with, that's when you'll want to pull it and "check" that things are still working.
Which file is "the" file? The "ca" file defined in your controller config file? Yes that's the one to update. (i'd probably backup what's there first 
)
identity:
cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/certs/ip-172-31-47-200-client.cert"
server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/certs/ip-172-31-47-200-server.chain.pem"
key: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/keys/ip-172-31-47-200-server.key"
ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/cas.pem"
1 Like
Hehe, this seems to work. After changing the cas.pem on the controller as well, I don't get the error for doing
ztx = openziti.load('test_identity.json')
anymore. It seems like somewhere the cas.pem on the controller is also used during runtime.
1 Like
Okay, looks like I was a little too fast assuming everything works now
Connections don't work and the controller shows for the routers:
connection error handler for [tls:myip] (x509): certificate signed by unknown authority
Do I have to adjust something with the routers as well?
I finally got this up and running! For anyone looking to upgrade from an older OpenZiti quickstart instance to a newer one, there are a few extra steps you’ll need to follow. Some of these might seem a bit counterintuitive, so I’ve outlined them below.
Key Points to Keep in Mind:
-
The cas.pem (CA bundle) file is exactly what it says: a bundle of CA certificates, not a chain. The order of CAs in this file doesn’t matter.
-
The cas.pem file is used to assign the appropriate CAs to newly created identities. Each identity manages its own CA store—system OS CA stores are not used.
-
In older quickstart versions, cas.pem also included intermediate certificates. In newer versions, it only contains the Root CA certificates. This means that any non-root certificates (those without "root" in their names) should be removed. Additionally, make sure that all Root CA certificates are included in the file.
-
After updating the cas.pem file to only contain the Root CA certificates, you may notice that communication between the Router and the Controller stops working. To fix this, you need to modify the client router certificate to include a full chain (excluding the Root CA). You can reference this file in the router YAML as shown below:
identity:
cert: "/home/ziti/.ziti/quickstart/zt/pki/routers/zt-edge-router/client.cert"
-
Similarly, communication between Identities and the Router will also break after updating the cas.pem file. To resolve this, edit the server router certificate to include a full chain (again, excluding the Root CA). This file is also referenced in the router YAML as:
identity:
server_cert: "/home/ziti/.ziti/quickstart/zt/pki/routers/zt-edge-router/server.cert"
1. Modify cas.pem to only contain (all) Root CA certs.
Remember backing up the quickstart directory!!
If you used the default quickstart, this should do the trick to create the new cas.pem:
echo "" > /your-quickstart-path/pki/cas.pem && find /your-quickstart-path/pki/ -type f -name "*root-ca.cert" -exec cat {} >> /your-quickstart-path/pki/cas.pem \;
2. Modify client router certificate to include a full chain
Again, a little tricky to do automatically. But done in 2 minutes:
openssl x509 -in pki/routers/zt-edge-router/client.cert -text | grep Issuer, then do the same for the issuer cert etc.
3. Modify server router certificate to include a full chain
This might be a little tricky to do automatically.
Instead of only the leaf certificate it should contain the full path up until before the root ca.
You can check the issuer subject and manually add the CA certs below. Do that until the issuer is a root ca:
openssl x509 -in pki/routers/zt-edge-router/server.cert -text | grep Issuer
4. Restart ziti-controller and ziti-router(s)
Enjoy.
2 Likes
