# Troubleshoot error - "invalid client certificate for api session" **URL:** \ **Category:** General Questions\ **Created:** [October 21, 2022, 12:37am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824 "2022-10-21T00:37:37Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 12:37am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/1 "2022-10-21T00:37:37Z")
I am getting this error on the edge router. ```auto Oct 21 00:32:33 ip-10-0-0-100.us-east-2.compute.internal ziti-router[2954309]: {"error":"invalid client certificate for api session","file":"github.com/openziti/channel/v2@v2.0.9/impl.go:126","func":"github.com/openziti/channel/v2.AcceptNextChannel.func1","level":"error","msg":"failure accepting channel edge with underlay u{classic}-\u003ei{noAQ}","time":"2022-10-21T00:32:33.028Z"} ``` What is the reason for this error? Is this coming while forming connections with other routers OR this is coming for identities which are trying to connect to this edge router? TIA. ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [October 21, 2022, 1:18am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/2 "2022-10-21T01:18:07Z")
I’m not 100% sure, but that appears to me to be an issue with the router accepting a connection form a client. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 1:19am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/3 "2022-10-21T01:19:54Z")
thats what I am thinking too but I am not sure how to troubleshoot it further. I tried using [Troubleshooting | Ziti](https://openziti.github.io/ziti/manage/troubleshooting.html#ziti-client-to-ziti-edge-router---data) and that didnt report any error. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 1:21am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/4 "2022-10-21T01:21:23Z")
Is there any healthcheck OR some api on edge router I can `curl` with endpoint client cert which might help to troubleshoot? ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [October 21, 2022, 1:28am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/5 "2022-10-21T01:28:20Z")
There’s no API for edge routers that I’m aware of/have used. This is an exception I’ve not personally experienced. I think you might be using a third party certificate? Is that correct? I’m only guessing here, but could you possibly have enrolled the identity, used it for a period, then regenerated the cert or anything like that? Maybe deleted the identity and reenrolled it or something like that while testing? It almost seems to me that the routers are out of sync somehow… Do you have any kind of extra information or steps we might take on our side to try to reproduce the situation? ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 1:37am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/6 "2022-10-21T01:37:02Z")
yes identity is bringing the third party certificate. the identity cert gets rotated every day but edge router config has `ca` set to correct trust bundle. if I do following ```auto verifyCertAgainstPool $edge_router_cert $identity_ca verifyCertAgainstPool $identity_cert $edge_router_ca ``` both return success. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 1:42am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/7 "2022-10-21T01:42:52Z")
Is this a valid assumption that edge router uses `.identity.ca` from its yaml configuration as the trust-store for all connections and similarly client uses `.id.ca` as the trust-store from its json identity file? ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [October 21, 2022, 1:50am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/8 "2022-10-21T01:50:11Z")
Yeah, I have a feeling there's some kind of session replication bug or 3rd party CA issue happening in here. I'm not "the expert" (or even "an expert") on this flow, but it seems like there might be a bug in there. @andrew.martinez will probably need to weigh in on that one > [@av-dev](#): > > edge router uses `.identity.ca` from its yaml configuration as the trust-store for all connections and similarly client uses `.id.ca` as the trust-store Yes that is consistent with my understanding. If I understand how the router enrollments work, the router `.identity.ca` is returned by the controller to the router during enrollment. I'm not certain how third party CAs are reflected at the edge routers. Again, I'm sure Andrew will be able to help there. Wish I could help more but I'll point this thread to him tomorrow and get his input. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 1:52am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/9 "2022-10-21T01:52:42Z")
Yeah if I check the router cas file it contains the ca bundle configured in controller config and it’s a correct bundle. And I believe same is given to endpoints on enrollment as well. So if I understand correctly , all 3 ca bundles are the same - controller , router and identity. Sure. Thank you so much for taking a stab at this. I will watch the thread for further updates. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 1:54am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/10 "2022-10-21T01:54:35Z")
I believe I have configured the correct bundle on the controller which contains all of the CA certs for my internal CA ( we have 5 independent CAs in 5 regions ) + the intermediate signing cert used for edge router enrollment. ---
**Author:** ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)\ **Post date:** [October 21, 2022, 2:09am UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/11 "2022-10-21T02:09:18Z")
I wouldn’t expect you to have to configure the CA bundle on the controller. It’s designed to allow you to upload them. Having to configure the bundle after the fact is not something I would expect to be necessary. But, Andrew is the person who will be able to more definitely answer that question. I’m as interested in the answer as you! 😃 ---
**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\ **Post date:** [October 21, 2022, 2:27pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/12 "2022-10-21T14:27:59Z")
> [@av-dev](#): > > What is the reason for this error? API Sessions are tied to certificates. Either the certificate used for authentication or an [API Session Certificate](https://openziti.github.io/ziti/security/authentication/api-session-certificates.html) created through the API Session. If a client attempts to connect to an Edge Router using a client certificate that is not tied to the API Session, you will get that error. You cannot mix and match client certificates and API Sessions. > [@av-dev](#): > > Is this coming while forming connections with other routers OR this is coming for identities which are trying to connect to this edge router? Clients connecting to an Edge Router w/ a client certificate not registered to the API Session. > [@av-dev](#): > > yes identity is bringing the third party certificate. the identity cert gets rotated every day The above quote sticks out to me. Rotated certificates w/o telling OpenZiti is a probable cause. Ziti can authenticate certificates in one of two fashions: either by fingerprint or some value in the certificate. The former is the default behavior. The latter only happens if you specify a claims property and embed either the identity `id` or `externalId` in that location. See: [https://openziti.github.io/ziti/security/authentication/third-party-cas.html#external-id--x509-claims](https://openziti.github.io/ziti/security/authentication/third-party-cas.html#external-id--x509-claims) Fingerprint matching is great for long-lived certificates and certificates that a Ziti network can re-issue. Short-lived certificates or certificates managed by 3rd party CAs are more suited to using claims within the certificate. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 2:34pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/13 "2022-10-21T14:34:01Z")
Where / How do I specify the externalId property? I created the CAs like below - ```auto ziti edge create ca myca-uw2 ~/myca-uw2.pem --autoca --auth --identity-name-format "[commonName]" --role-attributes "all-myca" --location "COMMON_NAME" --matcher "ALL" --parser "NONE" ``` ```auto ziti edge create ca myca-ue1 ~/myca-ue1.pem --autoca --auth --identity-name-format "[commonName]" --role-attributes "all-myca" --location "COMMON_NAME" --matcher "ALL" --parser "NONE" ``` And I was under the impression that when during the api session, externalId will be taken from cert CN. Is this correct? ---
**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\ **Post date:** [October 21, 2022, 2:39pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/14 "2022-10-21T14:39:42Z")
Looks like you are already using it by specifying `--location "COMMON_NAME" --matcher "ALL" --parser "NONE"`. Can you provide the output for: `ziti edge list cas -j`? There should only be public certificate information in there, but feel free to redact the certs if you wish. Also, does the client work? You were interested in the error message, but I didn’t see you state there was non-functionality. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 2:44pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/15 "2022-10-21T14:44:03Z")
```auto { "data": [ { "_links": { "jwt": { "href": "./cas/2Q9Pd6x6EiIYWNOdfZ1gWj/jwt" }, "self": { "href": "./cas/2Q9Pd6x6EiIYWNOdfZ1gWj" } }, "createdAt": "2022-10-14T22:41:52.668Z", "id": "2Q9Pd6x6EiIYWNOdfZ1gWj", "tags": {}, "updatedAt": "2022-10-14T22:52:47.172Z", "certPem": "", "externalIdClaim": { "index": 0, "location": "COMMON_NAME", "matcher": "ALL", "matcherCriteria": "", "parser": "NONE", "parserCriteria": "" }, "fingerprint": "9a594b6a6cd70acdeaf588ee98d4ea6bff9c6c01", "identityNameFormat": "[commonName]", "identityRoles": [ "all-myca" ], "isAuthEnabled": true, "isAutoCaEnrollmentEnabled": true, "isOttCaEnrollmentEnabled": false, "isVerified": true, "name": "myca-uw2", "verificationToken": "f9BQ4KzqY" }, { "_links": { "jwt": { "href": "./cas/5w6uGvtcjyMoDhRNRILvO5/jwt" }, "self": { "href": "./cas/5w6uGvtcjyMoDhRNRILvO5" } }, "createdAt": "2022-10-18T18:38:28.430Z", "id": "5w6uGvtcjyMoDhRNRILvO5", "tags": {}, "updatedAt": "2022-10-18T18:40:19.632Z", "certPem": "", "externalIdClaim": { "index": 0, "location": "COMMON_NAME", "matcher": "ALL", "matcherCriteria": "", "parser": "NONE", "parserCriteria": "" }, "fingerprint": "481d624fcf6f731b7d83849a20af5e88ae585647", "identityNameFormat": "[commonName]", "identityRoles": [ "all-myca" ], "isAuthEnabled": true, "isAutoCaEnrollmentEnabled": true, "isOttCaEnrollmentEnabled": false, "isVerified": true, "name": "myca-ue1", "verificationToken": "Pg7Kr0zuGW" }, { "_links": { "jwt": { "href": "./cas/6JTdlsJpsDszJ4JoygdZ96/jwt" }, "self": { "href": "./cas/6JTdlsJpsDszJ4JoygdZ96" } }, "createdAt": "2022-10-14T23:03:00.323Z", "id": "6JTdlsJpsDszJ4JoygdZ96", "tags": {}, "updatedAt": "2022-10-14T23:07:32.542Z", "certPem": "", "externalIdClaim": { "index": 0, "location": "COMMON_NAME", "matcher": "ALL", "matcherCriteria": "", "parser": "NONE", "parserCriteria": "" }, "fingerprint": "d9c1ade6f2fc37c2d889a7161e56859bcca6e9aa", "identityNameFormat": "[commonName]", "identityRoles": [ "all-myca" ], "isAuthEnabled": true, "isAutoCaEnrollmentEnabled": true, "isOttCaEnrollmentEnabled": false, "isVerified": true, "name": "myca-gq", "verificationToken": "nK6WRKzqY" } ], "meta": { "filterableFields": [ "isVerified", "verificationToken", "isAutoCaEnrollmentEnabled", "isSystem", "fingerprint", "updatedAt", "tags", "name", "isOttCaEnrollmentEnabled", "isAuthEnabled", "id", "createdAt" ], "pagination": { "limit": 10, "offset": 0, "totalCount": 3 } } } ``` ---
**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\ **Post date:** [October 21, 2022, 2:45pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/16 "2022-10-21T14:45:22Z")
> [@av-dev](#): > > Where / How do I specify the externalId property? I The `externalId` property is on `identities`. During `ziti edge create identity` (or update) it can be specified via `--external-id` ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 2:46pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/17 "2022-10-21T14:46:43Z")
client does not work. in ZAC it shows “API session” green and “edge router connected” as grey. I have used the same CA before and I didnt have any issue with certs being rotated every day. At that time the externalId part wasnt there though. That is something I added last week since I was interested in not creating identities per host. ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 2:48pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/18 "2022-10-21T14:48:00Z")
> [@andrew.martinez](#): > > The `externalId` property is on `identities`. During `ziti edge create identity` (or update) it can be specified via `--external-id` I guess this is where I might have messed up. I have used ziti edge enroll to create the identity first time and not an explicit create identity. ---
**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\ **Post date:** [October 21, 2022, 2:50pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/19 "2022-10-21T14:50:27Z")
So do the identities have any `externalId` values to map to your certificate’s common name values? `ziti edge list identities -j` will help you inspect them. If not, did you rely on auto-enrollment through the 3rd party ca or did you manually create the identities? ---
**Author:** ![av-dev](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@av-dev](https://openziti.discourse.group/u/av-dev)\ **Post date:** [October 21, 2022, 2:51pm UTC](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824/20 "2022-10-21T14:51:02Z")
Additional context what I am trying to achieve - [Ziti and spiffe - #14 by TheLumberjack](https://openziti.discourse.group/t/ziti-and-spiffe/733/14) We have a high churn of workloads and it would be unmanageable to keep track of each workload representing a separate identity. [Next page](https://openziti.discourse.group/t/troubleshoot-error-invalid-client-certificate-for-api-session/824.md?page=2)