# Python SDK throws "Controller not available" error

**URL:** <https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141>\
**Category:** Uncategorized\
**Created:** [February 24, 2024, 4:48pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141 "2024-02-24T16:48:49Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [September 11, 2024, 12:09pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/21 "2024-09-11T12:09:33Z")

</div>

Jumping into this thread for a second... Did you perhaps remove the "control plane CA"? aka the "ziti controller" CA from the bundle? the quickstarts were always created in a complex fashion, to demonstrate what is possible. As such - they ship with three different root CAs. One for the control plane, one for the edge apis and one for identities.

Between 0.26.8 and now, changes to the SDKs have changed the way the PKI that was generated works. Now, you must ensure the ca bundle contains root CAs only, and the full chain is used for the server certificates presented to complete the chain of trust.

It's possible the root ca for the control plane (router to router, router to controller, edge client to router) was removed. It's also possible the older router does not use the full chain.

You can test that your bundle is properly setup by doing two steps...

- download the bundle and unwrap it
- use the bundle with openssl to verify connectivity

In one copy/pastable block that looks like this (change ZITI\_EDGE\_CTRL\_ADVERTISED\_HOST\_PORT of course):

```auto
ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT=ec2-3-18-113-172.us-east-2.compute.amazonaws.com:8441

curl -sk https://"${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT}/.well-known/est/cacerts" \
  | openssl base64 -d \
  | openssl pkcs7 -inform DER -outform PEM -print_certs -text -out cacerts.pem

openssl s_client \
	-connect "${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT}" \
	-CAfile cacerts.pem < /dev/null \
	| grep Verify

```

You need to make sure you see: `Verify return code: 0 (ok)`. Assuming you do, I would expect everything to work.

Can we check your router's cert it's presenting to see if it's returning a chain? Dunno if @ekoby has anything else to add.

---

<div class="post-metadata">

**Author:** ![dmuensterer](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dmuensterer/32/2451_2.png) [@dmuensterer](https://openziti.discourse.group/u/dmuensterer)\
**Post date:** [September 11, 2024, 12:41pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/22 "2024-09-11T12:41:22Z")

</div>

Thank you Clint, I'll try that. I just yesterday followed your [PKI Ziti TV](https://www.youtube.com/watch?v=X_bvaZOAh34)

I'm still not 100% sure I understand correctly how the chain verification works with Ziti.

- For the CA bundle, certificate order doesn't matter, correct? It's not directly a chain file but rather the CAs are searched within that file, right?

- When the CA bundle only contains the Root Certs, where **exactly** on the server side are intermediary certs stored? Do I have to supply a chain with the leaf cert **AND** intermediary certs by every identity?

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [September 11, 2024, 12:48pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/23 "2024-09-11T12:48:28Z")

</div>

> [@dmuensterer](#):
>
> For the CA bundle, certificate order doesn't matter, correct?

correct..

> [@dmuensterer](#):
>
> where **exactly** on the server side are intermediary certs stored?

Assuming the quickstart was used and the locations haven't changed, all the PKI (keys, intermediates, roots, etc) will ALL be located at `$HOME/.ziti/quickstart/$(hostname)/pki`. Technically, you should open the controller config file and inspect the location, but it's 99.9% that they're there...

 ![image](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/2/2ecca1fe70512dd59e6ab93577ef41b332b28389.png)

> [@dmuensterer](#):
>
> Do I have to supply a chain with the leaf cert **AND** intermediary certs by every identity?

The identities need to have the CA bundle (which I expect they'll have) and the servers they connect to need to present the full chain, so identities would only need CAs in that situation. It should be find if the identities have the same certs as the servers present, hopefully that makes sense

---

<div class="post-metadata">

**Author:** ![dmuensterer](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dmuensterer/32/2451_2.png) [@dmuensterer](https://openziti.discourse.group/u/dmuensterer)\
**Post date:** [September 11, 2024, 12:48pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/24 "2024-09-11T12:48:28Z")

</div>

> [@TheLumberjack](#):
>
> You need to make sure you see: `Verify return code: 0 (ok)`. Assuming you do, I would expect everything to work.

That works:

```auto
openssl s_client \
        -connect "${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT}" \
        -CAfile modified_cas.pem < /dev/null \ 
        | grep Verify
Connecting to 46.38.242.110
depth=2 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=zt.mycompany.de-root-ca Root CA
verify return:1
depth=1 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=zt.mycompany.de-intermediate
verify return:1
depth=0 C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=zt.mycompany.de server certificate
verify return:1
DONE
Verify return code: 0 (ok)

```

But just for the sake of completeness I want to add that I didn't pull the cacerts from the controller but just composed it locally, since on the controller it's still the "old" CAs with the intermediary in it.  
Do I need to update this file right now as well for the testing or is its purpose just to supply newly generated identities with the CAs?

---

<div class="post-metadata">

**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\
**Post date:** [September 11, 2024, 1:01pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/25 "2024-09-11T13:01:57Z")

</div>

> [@dmuensterer](#):
>
> But just for the sake of completeness I want to add that I didn't pull the cacerts from the controller but just composed it locally,

Gotcha.. If you update the file the controller is configured with, that's when you'll want to pull it and "check" that things are still working.

> [@dmuensterer](#):
>
> Do I need to update this file right now as well for the testing or is its purpose just to supply newly generated identities with the CAs?

Which file is "the" file? The "ca" file defined in your controller config file? Yes that's the one to update. (i'd probably backup what's there first 🙂 )

```auto
identity:
  cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/certs/ip-172-31-47-200-client.cert"
  server_cert: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/certs/ip-172-31-47-200-server.chain.pem"
  key: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/keys/ip-172-31-47-200-server.key"
  ca: "/home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/cas.pem"

```

---

<div class="post-metadata">

**Author:** ![dmuensterer](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dmuensterer/32/2451_2.png) [@dmuensterer](https://openziti.discourse.group/u/dmuensterer)\
**Post date:** [September 11, 2024, 1:05pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/26 "2024-09-11T13:05:44Z")

</div>

Hehe, this seems to work. After changing the `cas.pem` on the controller as well, I don't get the error for doing

```auto
ztx = openziti.load('test_identity.json')

```

anymore. It seems like somewhere the `cas.pem` on the controller is also used during runtime.

---

<div class="post-metadata">

**Author:** ![dmuensterer](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dmuensterer/32/2451_2.png) [@dmuensterer](https://openziti.discourse.group/u/dmuensterer)\
**Post date:** [September 11, 2024, 3:13pm UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/27 "2024-09-11T15:13:52Z")

</div>

Okay, looks like I was a little too fast assuming everything works now 🙂  
Connections don't work and the controller shows for the routers:

```auto
connection error handler for [tls:myip] (x509): certificate signed by unknown authority

```

Do I have to adjust something with the routers as well?

---

<div class="post-metadata">

**Author:** ![dmuensterer](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/dmuensterer/32/2451_2.png) [@dmuensterer](https://openziti.discourse.group/u/dmuensterer)\
**Post date:** [September 12, 2024, 11:21am UTC](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141/28 "2024-09-12T11:21:16Z")

</div>

I finally got this up and running! For anyone looking to upgrade from an older OpenZiti quickstart instance to a newer one, there are a few extra steps you’ll need to follow. Some of these might seem a bit counterintuitive, so I’ve outlined them below.

### Key Points to Keep in Mind:

- The `cas.pem` (CA bundle) file is exactly what it says: a **bundle** of CA certificates, not a chain. The order of CAs in this file doesn’t matter.

- The `cas.pem` file is used to assign the appropriate CAs to newly created identities. Each identity manages its own CA store—system OS CA stores are not used.

- In older quickstart versions, `cas.pem` also included intermediate certificates. In newer versions, it only contains the Root CA certificates. This means that any non-root certificates (those without "root" in their names) should be removed. Additionally, make sure that **all** Root CA certificates are included in the file.

- After updating the `cas.pem` file to only contain the Root CA certificates, you may notice that communication between the Router and the Controller stops working. To fix this, you need to modify the client router certificate to include a full chain (excluding the Root CA). You can reference this file in the router YAML as shown below:

- Similarly, communication between Identities and the Router will also break after updating the cas.pem file. To resolve this, edit the server router certificate to include a full chain (again, excluding the Root CA). This file is also referenced in the router YAML as:

## 1. Modify cas.pem to only contain (all) Root CA certs.

**Remember backing up the quickstart directory!!**  
If you used the default quickstart, this should do the trick to create the new `cas.pem`:

```auto
echo "" > /your-quickstart-path/pki/cas.pem && find /your-quickstart-path/pki/ -type f -name "*root-ca.cert" -exec cat {} >> /your-quickstart-path/pki/cas.pem \;

```

## 2. Modify client router certificate to include a full chain

Again, a little tricky to do automatically. But done in 2 minutes:  
`openssl x509 -in pki/routers/zt-edge-router/client.cert -text | grep Issuer`, then do the same for the issuer cert etc.

## 3. Modify server router certificate to include a full chain

This might be a little tricky to do automatically.  
Instead of only the leaf certificate it should contain the full path up until before the root ca.  
You can check the issuer subject and manually add the CA certs below. Do that until the issuer is a root ca:  
`openssl x509 -in pki/routers/zt-edge-router/server.cert -text | grep Issuer`

## 4. Restart ziti-controller and ziti-router(s)

Enjoy.

[Previous page](https://openziti.discourse.group/t/python-sdk-throws-controller-not-available-error/2141.md?page=1)