Python SDK throws "Controller not available" error

Hi,

we've been trying to use the Python SDK for the first time but can't get past the following error:

(98264)[        0.084]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -53(software caused connection abort)
(98264)[        0.084]   ERROR ziti-sdk:ziti.c:1668 version_cb() ztx[0] failed to get controller version from https://zt.company.de:8441/edge/client/v1 CONTROLLER_UNAVAILABLE(software caused connection abort)
(98264)[        0.084]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.company.de] request failed: -53(software caused connection abort)
2024-02-24 17:42:05,332 - ERROR - could not find identity file: ctx is not a valid python void pointer type
Traceback (most recent call last):
  File "/Users/user/Projects/company/use-cases/export.py", line 411, in <module>
    asyncio.run(main(dry_run=args.dry_run))
  File "/Users/user/.pyenv/versions/3.11.1/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.11.1/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.11.1/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/user/Projects/company/use-cases/export.py", line 363, in main
    monkey_patch_openziti(deployment)
  File "/Users/user/Projects/company/use-cases/export.py", line 102, in monkey_patch_openziti
    raise e
  File "/Users/user/Projects/company/use-cases/export.py", line 99, in monkey_patch_openziti
    zitiContext = openziti.load('identity.json')
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.11.1/lib/python3.11/site-packages/openziti/context.py", line 87, in load_identity
    return ZitiContext.from_path(path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.11.1/lib/python3.11/site-packages/openziti/context.py", line 77, in from_path
    return cls(zitilib.load(path))
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.11.1/lib/python3.11/site-packages/openziti/context.py", line 27, in __init__
    raise TypeError("ctx is not a valid python void pointer type")
TypeError: ctx is not a valid python void pointer type

Here's the code we're using:

    try:
        zitiContext = openziti.load('identity.json')
    except Exception as e:
        logging.error(f'could not find identity file: {e}')
        raise e
 
    cfg = dict(
        ztx=zitiContext,
        service='Test_Github_Action',
    )
    openziti.monkeypatch(bindings={deployment.endpoint: cfg})

We are using v0.8.1 and we're certain that the controller is in fact available by the host because Ziti Desktop Edge works flawlessly on the same device.
We've enrolled the JWT on the controller via ziti edge enroll and copied the JSON.

What are we missing here?

1 Like

Since the error says it can't find the identity file, I would start by giving it a full path. It could also be a permissions issue, but that would be less likely.

Thanks, but it can get the Controller URL from the identity file right?
This was a sign for me that the JSON is being opened correctly. The could not find identity file is just the default error message from the quickstart.

was this Ziti instance created by an older version of quickstart?

It is possible that CA bundle is not correct.
Python SDK is using OpenSSL for TLS connectivity, ZDE is using mbedTLS.

OpenSSL is more strict wrt certificate validation, and older versions of quickstart produced incomplete CA bundles

@dmuensterer Can you try with an identity from a fresh quickstart instance?

Aha, that’s interesting. I can try that. The QuickStart was from September 2022.

I couldn't yet setup a complete new quickstart.
We didn't pursue this issue with the Python SDK further, but we just now happened to see the same issue with the ziti-node-sdk:

[        0.033] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:272 _ziti_init(): initializing
[        0.033] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:302 _ziti_init(): config_file_name: /Users/username/GIT/docs/.ziti-identity.json
(34988)[        0.000]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=5/VERBOSE
(34988)[        0.000]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.36.9 @d336721(HEAD) starting at (2024-08-21T10:33:35.382)
(34988)[        0.000]   DEBUG ziti-sdk:config.c:58 ziti_load_config() trying to load config from file[/Users/username/GIT/docs/.ziti-identity.json]
[        0.033] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:342 _ziti_init(): ziti_load_config => 0
[        0.033] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:346 _ziti_init(): ziti_context_init => 0
[        0.033] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:358 _ziti_init(): ziti_context_set_options => 0
[        0.037] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:362 _ziti_init(): ziti_context_run => 0
(34988)[        0.006]    INFO ziti-sdk:ziti.c:455 ziti_init_async() ztx[0] using tlsuv[v0.28.4], tls[OpenSSL 3.1.4 24 Oct 2023]
(34988)[        0.006]    INFO ziti-sdk:ziti.c:456 ziti_init_async() ztx[0] Loading ziti context with controller[https://zt.mycompany.de:8441/edge/client/v1]
(34988)[        0.006]   DEBUG ziti-sdk:ziti_ctrl.c:415 ziti_ctrl_init() ctrl[zt.mycompany.de] ziti controller client initialized
(34988)[        0.006]   DEBUG ziti-sdk:ziti.c:478 ziti_init_async() ztx[0] using metrics interval: 6
(34988)[        0.006] VERBOSE ziti-sdk:ziti_ctrl.c:143 start_request() ctrl[zt.mycompany.de] starting GET[/version]
(34988)[        0.006]   DEBUG ziti-sdk:ziti.c:280 ziti_set_unauthenticated() ztx[0] setting api_session_state[0] to 0
(34988)[        0.006]   DEBUG ziti-sdk:ziti_ctrl.c:254 ziti_ctrl_clear_api_session() ctrl[zt.mycompany.de] clearing api session token for ziti_controller
(34988)[        0.006]   DEBUG ziti-sdk:ziti.c:986 ziti_re_auth() ztx[0] re-auth executing, transitioning to unauthenticated
(34988)[        0.006]   DEBUG ziti-sdk:ziti.c:280 ziti_set_unauthenticated() ztx[0] setting api_session_state[0] to 0
(34988)[        0.006]   DEBUG ziti-sdk:ziti_ctrl.c:254 ziti_ctrl_clear_api_session() ctrl[zt.mycompany.de] clearing api session token for ziti_controller
(34988)[        0.006]   DEBUG ziti-sdk:ziti.c:322 is_api_session_expired() ztx[0] is_api_session_expired[TRUE] - api_session is null
(34988)[        0.006]    INFO ziti-sdk:ziti.c:934 ziti_re_auth_with_cb() ztx[0] starting to re-auth with ctrl[https://zt.mycompany.de:8441/edge/client/v1] api_session_status[0] api_session_expired[TRUE]
(34988)[        0.006]   DEBUG ziti-sdk:ziti.c:273 ziti_set_auth_started() ztx[0] setting api_session_state[0] to 1
(34988)[        0.006]   DEBUG ziti-sdk:ziti.c:357 ziti_stop_api_session_refresh() ztx[0] ziti_stop_api_session_refresh: stopping api session refresh
(34988)[        0.006] VERBOSE ziti-sdk:ziti_ctrl.c:143 start_request() ctrl[zt.mycompany.de] starting POST[/authenticate?method=cert]
[        0.043] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_set_log_level.c:45 _ziti_set_log_level(): js_log_level: 5
(34988)[        0.007]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=5/VERBOSE
[        0.043] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:272 _ziti_init(): initializing
[        0.043] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:302 _ziti_init(): config_file_name: /Users/username/GIT/docs/.ziti-identity.json
(34988)[        0.007]    INFO ziti-sdk:utils.c:199 ziti_log_set_level() set log level: root=5/VERBOSE
(34988)[        0.007]    INFO ziti-sdk:utils.c:170 ziti_log_init() Ziti C SDK version 0.36.9 @d336721(HEAD) starting at (2024-08-21T10:33:35.392)
(34988)[        0.007]   DEBUG ziti-sdk:config.c:58 ziti_load_config() trying to load config from file[/Users/username/GIT/docs/.ziti-identity.json]
[        0.048] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:342 _ziti_init(): ziti_load_config => 0
[        0.048] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:346 _ziti_init(): ziti_context_init => 0
[        0.048] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:358 _ziti_init(): ziti_context_set_options => 0
[        0.050] DEBUG   ziti-sdk-nodejs//Users/runner/work/ziti-sdk-nodejs/ziti-sdk-nodejs/src/ziti_init.c:362 _ziti_init(): ziti_context_run => 0
(34988)[        0.021]    INFO ziti-sdk:ziti.c:455 ziti_init_async() ztx[1] using tlsuv[v0.28.4], tls[OpenSSL 3.1.4 24 Oct 2023]
(34988)[        0.021]    INFO ziti-sdk:ziti.c:456 ziti_init_async() ztx[1] Loading ziti context with controller[https://zt.mycompany.de:8441/edge/client/v1]
(34988)[        0.021]   DEBUG ziti-sdk:ziti_ctrl.c:415 ziti_ctrl_init() ctrl[zt.mycompany.de] ziti controller client initialized
(34988)[        0.021]   DEBUG ziti-sdk:ziti.c:478 ziti_init_async() ztx[1] using metrics interval: 6
(34988)[        0.021] VERBOSE ziti-sdk:ziti_ctrl.c:143 start_request() ctrl[zt.mycompany.de] starting GET[/version]
(34988)[        0.021]   DEBUG ziti-sdk:ziti.c:280 ziti_set_unauthenticated() ztx[1] setting api_session_state[0] to 0
(34988)[        0.021]   DEBUG ziti-sdk:ziti_ctrl.c:254 ziti_ctrl_clear_api_session() ctrl[zt.mycompany.de] clearing api session token for ziti_controller
(34988)[        0.021]   DEBUG ziti-sdk:ziti.c:986 ziti_re_auth() ztx[1] re-auth executing, transitioning to unauthenticated
(34988)[        0.021]   DEBUG ziti-sdk:ziti.c:280 ziti_set_unauthenticated() ztx[1] setting api_session_state[0] to 0
(34988)[        0.021]   DEBUG ziti-sdk:ziti_ctrl.c:254 ziti_ctrl_clear_api_session() ctrl[zt.mycompany.de] clearing api session token for ziti_controller
(34988)[        0.021]   DEBUG ziti-sdk:ziti.c:322 is_api_session_expired() ztx[1] is_api_session_expired[TRUE] - api_session is null
(34988)[        0.021]    INFO ziti-sdk:ziti.c:934 ziti_re_auth_with_cb() ztx[1] starting to re-auth with ctrl[https://zt.mycompany.de:8441/edge/client/v1] api_session_status[0] api_session_expired[TRUE]
(34988)[        0.021]   DEBUG ziti-sdk:ziti.c:273 ziti_set_auth_started() ztx[1] setting api_session_state[0] to 1
(34988)[        0.021]   DEBUG ziti-sdk:ziti.c:357 ziti_stop_api_session_refresh() ztx[1] ziti_stop_api_session_refresh: stopping api session refresh
(34988)[        0.021] VERBOSE ziti-sdk:ziti_ctrl.c:143 start_request() ctrl[zt.mycompany.de] starting POST[/authenticate?method=cert]
(34988)[        0.093]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[zt.mycompany.de] request failed: -53(software caused connection abort)
(34988)[        0.093]   ERROR ziti-sdk:ziti.c:1696 version_cb() ztx[0] failed to get controller version from https://zt.mycompany.de:8441/edge/client/v1 CONTROLLER_UNAVAILABLE(software caused connection abort)
(34988)[        0.093]   ERROR ziti-sdk:ziti_ctrl.c:164 ctrl_resp_cb() ctrl[zt.mycompany.de] request failed: -53(software caused connection abort)
(34988)[        0.093]    WARN ziti-sdk:ziti.c:1625 api_session_cb() ztx[0] failed to get api session from ctrl[https://zt.mycompany.de:8441/edge/client/v1] api_session_state[1] CONTROLLER_UNAVAILABLE[-16] software caused connection abort
(34988)[        0.093]   DEBUG ziti-sdk:ziti.c:1666 api_session_cb() ztx[0] unhandled error, setting api_session_timer to 5s
(34988)[        0.093]   DEBUG ziti-sdk:ziti.c:280 ziti_set_unauthenticated() ztx[0] setting api_session_state[1] to 0
(34988)[        0.093]   DEBUG ziti-sdk:ziti_ctrl.c:254 ziti_ctrl_clear_api_session() ctrl[zt.mycompany.de] clearing api session token for ziti_controller

Do we really need to setup a whole new Ziti Infrastructure to get the SDKs working?
Is there a way to generate "new" identitites? We're using Ziti v1.1.5

1 Like

Hi! Any help would be highly appreciated.
The ziti quickstart is from September 2022, is this a problem?
If not, any ideas what we're doing wrong?

If it helps I have just updated my QuickStart from 0.28.1 to 1.1.10 yesterday. Did it in 3 hops: 0.28.3 (environment variable breaking change) , to 1.1.5 (UID change) and then to 1.1.10 (required SPIFFE trustDomain variable as cert does not have it). I have all the steps and files if you are interested.

Not sure if you can just take the bolt db and move it. It is what I did anyway.

2 Likes

There are many variables. I would start fresh. Now the ziti CLI has a built in quickstart you can use to create temporary networks for development and testing, at least.

Run:

ziti edge quickstart

The default password is "admin". There's a Docker version of this all-in-one quickstart.

If you wish to preserve state then provide a path to a directory like --home ./config.

For long lived deployments check out the guides for Linux, Docker, and Kubernetes.

You asked if it's necessary to run Ziti infrastructure. That's probably best because it gives you admin control over policies, configs, etc. that are part of SDK implementation.

Really not sure what we're doing wrong here:
We are running a very recent version of the ziti controller, Ziti v1.1.5 but neither of the SDKs seems to work for us:

import openziti

# load ziti context, provide full path to identity file if needed
ztx = openziti.load('test_identity.json')

This already results in the following error:

(3510143)[        0.025]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.mycompany.de] request failed: -103(software caused connection abort)
(3510143)[        0.025]   ERROR ziti-sdk:ziti.c:1668 version_cb() ztx[0] failed to get controller version from https://zt.mycompany.de:8441/edge/client/v1 CONTROLLER_UNAVAILABLE(software caused connection abort)
(3510143)[        0.025]   ERROR ziti-sdk:ziti_ctrl.c:162 ctrl_resp_cb() ctrl[zt.mycompany.de] request failed: -103(software caused connection abort)
Traceback (most recent call last):
  File "/tmp/python-ziti-test/ziti.py", line 5, in <module>
    ztx = openziti.load('test_identity.json')
  File "/home/dmuensterer/.local/lib/python3.9/site-packages/openziti/context.py", line 87, in load_identity
    return ZitiContext.from_path(path)
  File "/home/dmuensterer/.local/lib/python3.9/site-packages/openziti/context.py", line 77, in from_path
    return cls(zitilib.load(path))
  File "/home/dmuensterer/.local/lib/python3.9/site-packages/openziti/context.py", line 27, in __init__
    raise TypeError("ctx is not a valid python void pointer type")
TypeError: ctx is not a valid python void pointer type

Here's the result of a manual curl -k https://zt.mycompany.de:8441/edge/client/v1

{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://zt.mycompany.de:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://zt.mycompany.de:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://zt.mycompany.de:18441/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2024-07-02T18:20:02Z","capabilities":[],"revision":"aec0d3b9acfb","runtimeVersion":"go1.22.4","version":"v1.1.5"},"meta":{}}

test_identity.json:

{
    "ztAPI": "https://zt.mycompany.de:8441/edge/client/v1",
    "ztAPIs": null,
    "configTypes": null,
    "id": {
        "key": "pem:-----BEGIN RSA PRIVATE KEY-----\nREDACTED\n-----END RSA PRIVATE KEY-----\n",
        "cert": "pem:-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----\n",
        "ca": "pem:-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----\n"
    },
    "enableHa": false
}

Is there an easy way for me to find out if it's incorrect? We have lots and lots of identities deployed so replacing every identity is not an easy option.
Can I transform the ca-bundle to be correct?

That would be highly appreciated.
We are already running 1.1.5, but didn't do any changes to the CA bundle. Did you?
After quite some testing and setting up a second test environment, we got the SDKs running with a new quickstart.

But I'm not sure if/how I can upgrade/transform the CA bundles to the "new" format.
Any ideas? It would be really unfortunate if we had to replace hundreds of identities.

the most common issue with old quickstart identities is intermediate certificates in the CA bundle. Try removing them and see if that works. Once you have the CA bundle that works, it should work for all identities.

I guess you can script the update of all identities with some jq-fu -- something like this

$ jq --rawfile new_ca new_ca.pem '.id.ca=$new_ca' orig_identity.json > new_identity.json

You can also get the new CA bundle directly from controller:

curl -k https://<your-ziti-controller-host-and-port>/.well-known/est/cacerts | base64 -d | openssl pkcs7 -inform DER -print_certs
1 Like

Thank you! I’ll try that

Huh, interestingly enough with a new quickstart there are quite a lot more CA certificates which makes me unsure which intermediate CA I don't need anymore or if I have to create other CAs as well:

OLD quickstart:

zt-intermediate

    Subject: CN=zt-intermediate
    Issuer: CN=zt-root-ca Root CA

zt-signing-intermediate

    Subject: CN=zt-signing-intermediate
    Issuer: CN=zt-signing-intermediate_spurious_intermediate

zt.mycompany.de-intermediate

    Subject: CN=zt.mycompany.de-intermediate
    Issuer: CN=zt.mycompany.de-root-ca Root CA

NEW quickstart:

zt-test-signing-root-ca Root CA

    Subject: CN=zt-test-signing-root-ca Root CA
    Issuer: CN=zt-test-signing-root-ca Root CA

zt-test-edge-controller-root-ca Root CA

    Subject: CN=zt-test-edge-controller-root-ca Root CA
    Issuer: CN=zt-test-edge-controller-root-ca Root CA

zt-test-signing-intermediate

    Subject: CN=zt-test-signing-intermediate
    Issuer: CN=zt-test-signing-intermediate_grandparent_intermediate

zt-test-signing-intermediate_grandparent_intermediate

    Subject: CN=zt-test-signing-intermediate_grandparent_intermediate
    Issuer: CN=zt-test-signing-root-ca Root CA

zt-test-root-ca Root CA
    Subject: CN=zt-test-root-ca Root CA
    Issuer: CN=zt-test-root-ca Root CA

Just to confirm:
The problem is also with newly created identities by the controller v1.1.5: It seems as if the complete CA bundle on the controller side as well has changed quite a bit.

what flavor of quickstart are you using?

Using the "Host OpenZiti Anywhere" quickstart.
I deployed the quickstart when v0.26.8 was current :slight_smile: wow, how time flies

I was able to adjust the CA bundle so that the communication to the controller works.
However, now with the adjusted bundle I'm getting another error:

(3781)[        2.345]   ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[1] failed to connect to ER[zt-edge-router] [-53/software caused connection abort]
(3781)[        2.381]   ERROR ziti-sdk:channel.c:861 on_channel_connect_internal() ch[0] failed to connect to ER[zt-router-1.mycompany.ziti] [-53/software caused connection abort]

Do I need to change something for the router identities as well?
The router log says

Sep 11 13:47:23 zt-router-1 ziti[633094]: {"_context":"tls:0.0.0.0:8442","error":"local error: tls: bad record MAC","file":"github.com/openziti/transport/v2@v2.0.138/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedList>