Troubleshoot error - "invalid client certificate for api session"

Support General Questions
1

I am getting this error on the edge router.

Oct 21 00:32:33 ip-10-0-0-100.us-east-2.compute.internal ziti-router[2954309]: {"error":"invalid client certificate for api session","file":"github.com/openziti/channel/v2@v2.0.9/impl.go:126","func":"github.com/openziti/channel/v2.AcceptNextChannel.func1","level":"error","msg":"failure accepting channel edge with underlay u{classic}-\u003ei{noAQ}","time":"2022-10-21T00:32:33.028Z"}

What is the reason for this error? Is this coming while forming connections with other routers OR this is coming for identities which are trying to connect to this edge router?

TIA.

2

I’m not 100% sure, but that appears to me to be an issue with the router accepting a connection form a client.

3

thats what I am thinking too but I am not sure how to troubleshoot it further. I tried using Troubleshooting | Ziti and that didnt report any error.

4

Is there any healthcheck OR some api on edge router I can curl with endpoint client cert which might help to troubleshoot?

5

There’s no API for edge routers that I’m aware of/have used. This is an exception I’ve not personally experienced. I think you might be using a third party certificate? Is that correct?

I’m only guessing here, but could you possibly have enrolled the identity, used it for a period, then regenerated the cert or anything like that? Maybe deleted the identity and reenrolled it or something like that while testing?

It almost seems to me that the routers are out of sync somehow…

Do you have any kind of extra information or steps we might take on our side to try to reproduce the situation?

6

yes identity is bringing the third party certificate. the identity cert gets rotated every day but edge router config has ca set to correct trust bundle.
if I do following

verifyCertAgainstPool $edge_router_cert $identity_ca
verifyCertAgainstPool $identity_cert $edge_router_ca

both return success.

7

Is this a valid assumption that edge router uses .identity.ca from its yaml configuration as the trust-store for all connections and similarly client uses .id.ca as the trust-store from its json identity file?

8

Yeah, I have a feeling there's some kind of session replication bug or 3rd party CA issue happening in here. I'm not "the expert" (or even "an expert") on this flow, but it seems like there might be a bug in there. @andrew.martinez will probably need to weigh in on that one

Yes that is consistent with my understanding. If I understand how the router enrollments work, the router .identity.ca is returned by the controller to the router during enrollment. I'm not certain how third party CAs are reflected at the edge routers. Again, I'm sure Andrew will be able to help there.

Wish I could help more but I'll point this thread to him tomorrow and get his input.

9

Yeah if I check the router cas file it contains the ca bundle configured in controller config and it’s a correct bundle. And I believe same is given to endpoints on enrollment as well. So if I understand correctly , all 3 ca bundles are the same - controller , router and identity.

Sure. Thank you so much for taking a stab at this. I will watch the thread for further updates.

10

I believe I have configured the correct bundle on the controller which contains all of the CA certs for my internal CA ( we have 5 independent CAs in 5 regions ) + the intermediate signing cert used for edge router enrollment.

11

I wouldn’t expect you to have to configure the CA bundle on the controller. It’s designed to allow you to upload them. Having to configure the bundle after the fact is not something I would expect to be necessary. But, Andrew is the person who will be able to more definitely answer that question. I’m as interested in the answer as you! :smiley:

12

API Sessions are tied to certificates. Either the certificate used for authentication or an API Session Certificate created through the API Session. If a client attempts to connect to an Edge Router using a client certificate that is not tied to the API Session, you will get that error. You cannot mix and match client certificates and API Sessions.

Clients connecting to an Edge Router w/ a client certificate not registered to the API Session.

The above quote sticks out to me. Rotated certificates w/o telling OpenZiti is a probable cause. Ziti can authenticate certificates in one of two fashions: either by fingerprint or some value in the certificate. The former is the default behavior. The latter only happens if you specify a claims property and embed either the identity id or externalId in that location. See: https://openziti.github.io/ziti/security/authentication/third-party-cas.html#external-id--x509-claims

Fingerprint matching is great for long-lived certificates and certificates that a Ziti network can re-issue. Short-lived certificates or certificates managed by 3rd party CAs are more suited to using claims within the certificate.

13

Where / How do I specify the externalId property? I created the CAs like below -

ziti edge create ca myca-uw2 ~/myca-uw2.pem --autoca --auth --identity-name-format "[commonName]" --role-attributes "all-myca" --location "COMMON_NAME" --matcher "ALL" --parser "NONE"

ziti edge create ca myca-ue1 ~/myca-ue1.pem --autoca --auth --identity-name-format "[commonName]" --role-attributes "all-myca" --location "COMMON_NAME" --matcher "ALL" --parser "NONE"

And I was under the impression that when during the api session, externalId will be taken from cert CN. Is this correct?

14

Looks like you are already using it by specifying --location "COMMON_NAME" --matcher "ALL" --parser "NONE".

Can you provide the output for: ziti edge list cas -j?

There should only be public certificate information in there, but feel free to redact the certs if you wish.

Also, does the client work? You were interested in the error message, but I didn’t see you state there was non-functionality.

15 
{
    "data": [
        {
            "_links": {
                "jwt": {
                    "href": "./cas/2Q9Pd6x6EiIYWNOdfZ1gWj/jwt"
                },
                "self": {
                    "href": "./cas/2Q9Pd6x6EiIYWNOdfZ1gWj"
                }
            },
            "createdAt": "2022-10-14T22:41:52.668Z",
            "id": "2Q9Pd6x6EiIYWNOdfZ1gWj",
            "tags": {},
            "updatedAt": "2022-10-14T22:52:47.172Z",
            "certPem": "<CA_CERT_PEM>",
            "externalIdClaim": {
                "index": 0,
                "location": "COMMON_NAME",
                "matcher": "ALL",
                "matcherCriteria": "",
                "parser": "NONE",
                "parserCriteria": ""
            },
            "fingerprint": "9a594b6a6cd70acdeaf588ee98d4ea6bff9c6c01",
            "identityNameFormat": "[commonName]",
            "identityRoles": [
                "all-myca"
            ],
            "isAuthEnabled": true,
            "isAutoCaEnrollmentEnabled": true,
            "isOttCaEnrollmentEnabled": false,
            "isVerified": true,
            "name": "myca-uw2",
            "verificationToken": "f9BQ4KzqY"
        },
        {
            "_links": {
                "jwt": {
                    "href": "./cas/5w6uGvtcjyMoDhRNRILvO5/jwt"
                },
                "self": {
                    "href": "./cas/5w6uGvtcjyMoDhRNRILvO5"
                }
            },
            "createdAt": "2022-10-18T18:38:28.430Z",
            "id": "5w6uGvtcjyMoDhRNRILvO5",
            "tags": {},
            "updatedAt": "2022-10-18T18:40:19.632Z",
            "certPem": "<CA_CERT_PEM>",
            "externalIdClaim": {
                "index": 0,
                "location": "COMMON_NAME",
                "matcher": "ALL",
                "matcherCriteria": "",
                "parser": "NONE",
                "parserCriteria": ""
            },
            "fingerprint": "481d624fcf6f731b7d83849a20af5e88ae585647",
            "identityNameFormat": "[commonName]",
            "identityRoles": [
                "all-myca"
            ],
            "isAuthEnabled": true,
            "isAutoCaEnrollmentEnabled": true,
            "isOttCaEnrollmentEnabled": false,
            "isVerified": true,
            "name": "myca-ue1",
            "verificationToken": "Pg7Kr0zuGW"
        },
        {
            "_links": {
                "jwt": {
                    "href": "./cas/6JTdlsJpsDszJ4JoygdZ96/jwt"
                },
                "self": {
                    "href": "./cas/6JTdlsJpsDszJ4JoygdZ96"
                }
            },
            "createdAt": "2022-10-14T23:03:00.323Z",
            "id": "6JTdlsJpsDszJ4JoygdZ96",
            "tags": {},
            "updatedAt": "2022-10-14T23:07:32.542Z",
            "certPem": "<CA_CERT_PEM>",
            "externalIdClaim": {
                "index": 0,
                "location": "COMMON_NAME",
                "matcher": "ALL",
                "matcherCriteria": "",
                "parser": "NONE",
                "parserCriteria": ""
            },
            "fingerprint": "d9c1ade6f2fc37c2d889a7161e56859bcca6e9aa",
            "identityNameFormat": "[commonName]",
            "identityRoles": [
                "all-myca"
            ],
            "isAuthEnabled": true,
            "isAutoCaEnrollmentEnabled": true,
            "isOttCaEnrollmentEnabled": false,
            "isVerified": true,
            "name": "myca-gq",
            "verificationToken": "nK6WRKzqY"
        }
    ],
    "meta": {
        "filterableFields": [
            "isVerified",
            "verificationToken",
            "isAutoCaEnrollmentEnabled",
            "isSystem",
            "fingerprint",
            "updatedAt",
            "tags",
            "name",
            "isOttCaEnrollmentEnabled",
            "isAuthEnabled",
            "id",
            "createdAt"
        ],
        "pagination": {
            "limit": 10,
            "offset": 0,
            "totalCount": 3
        }
    }
}
16

The externalId property is on identities. During ziti edge create identity (or update) it can be specified via --external-id

17

client does not work. in ZAC it shows “API session” green and “edge router connected” as grey.

I have used the same CA before and I didnt have any issue with certs being rotated every day. At that time the externalId part wasnt there though. That is something I added last week since I was interested in not creating identities per host.

18

I guess this is where I might have messed up. I have used ziti edge enroll to create the identity first time and not an explicit create identity.

19

So do the identities have any externalId values to map to your certificate’s common name values?

ziti edge list identities -j will help you inspect them.

If not, did you rely on auto-enrollment through the 3rd party ca or did you manually create the identities?

20

Additional context what I am trying to achieve - Ziti and spiffe - #14 by TheLumberjack We have a high churn of workloads and it would be unmanageable to keep track of each workload representing a separate identity.