Ziti edge enrol identity not inserting CA bundle

Ziti Overlay
21

Here are the results for the next test


        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=instance-20220416-1603-signing-intermediate
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=instance-20220416-1603 server certificate
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=instance-20220416-1603-intermediate
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=instance-20220416-1603-signing-intermediate
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=168.138.10.79-intermediate
        Subject: C=US, L=Charlotte, O=NetFoundry, OU=ADV-DEV, CN=168.138.10.79-intermediate
22

Your assistance with this is greatly appreciated… it certainly helps with understanding how all of the certificates work.

For instance… I never really understood what the signing certificate was used for… then it clicked…as its what is used to enrol identities…

It would be great if there was a resource that you can point me to so that I can better understand the certificate hierarchy… and the purpose of each level… along with… the controls required for each layer

For instance… my understanding is that the root certificate unlocks everything… the intermediate certificates is used to delegate signing… and signing certificates to facilitate mutual trust.

Let me know if that makes any sense… or where I can go to better understand and fill in a few more gaps.

23

You're welcome. I'm personally annoyed that it's not working for you! This is a particularly difficult area to troubleshoot, your patience and tenacity is also appreciated.

Lol yeah, FOR SURE. it's another one of those things that just needs to get done, we don't have any lack for things to do, but a guide and explainer of ALL the things the quickstart does, and why, is an issue I filled last week
fully explain the quickstart · Issue #879 · openziti/ziti · GitHub.

Eh, kinda... You need "a root of trust", but that root doesn't have to be the same root for the three basic parts of a ziti solution: the network pki, the edge (controller api) pki, the edge (clients) pki. The current quickstart goes out of it's way to make these three different roots of trust... There's more to it but maybe that will help.

Here we are trying to figure out why you're not getting anything in your CA field. It's really odd. All our tests seem to indicate that things are working fine. I think we are at the point where I need an identity and I need to debug the code. Could you make me like, 10 identities and directly message them to me? I'll step through the debugger and see what that shows me? I'll need access to your controller from the debugger, would you be willing to open my ip to access your controller? I'll follow up in dm

24

If you haven’t seen some of Andrew’s latest content, it may be useful - Integrate OpenZiti with JWTs, PKIs for seamless service authentication

1 Like
25

That’s a great article… thanks for sharing

26

@markamind - there’s “definitely something strange” happening for you. For the others, I have access to @markamind’s controller and I’ve been making/deleting the same identity trying to figure out what/where/why something is going wrong. it seems to ONLY fail from the controller. Every other machine I’ve enrolled the identity on works.

I’ll keep at this, but probably not till tomorrow. I’d like to retain access to your controller to continue testing. I’ll let you know when I work through ‘the problem’

1 Like
27

Thanks for looking into this.. that is a very interesting observation.. I cannot think of anything specific that I did on that server that would cause this to happen..

I was thinking did I change a port.. modify something on the VCN.. I don't think so.. as otherwise.. it would not work at all

28

Would you try that test yourself? Could you create an identity for yourself then try to enroll it anywhere else? I tried it on 2 different machines but, maybe I’m doing it differently. If you confirm you can make the identity then move it off that controller and enroll it, it’d be useful.

1 Like
29

Will do… I will revert back later today…

30

I’ve gotten down to the bottom of the issue. At some point you have added the CA for the controller into your OS’s trust store. Looking at your /etc/ssl/certs folder, I can see that you have (somehow, not sure how) updated some symlinks in there:

ll /etc/ssl/certs/
total 0
lrwxrwxrwx. 1 root root 49 Sep 13 08:35 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root 55 Sep 13 08:35 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

The proof is in a simple curl. Since ziti deploys a PKI of its own, a curl without the -k flag should fail, like it does from my local computer:

curl from clint’s computer:

curl https://168.138.10.79:8441/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

curl from @markamind’s controller:

curl https://168.138.10.79:8441/
{"data":{"apiVersions":{"edge":{"v1":{"apiBaseUrls":["https://168.138.10.79:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-client":{"v1":{"apiBaseUrls":["https://168.138.10.79:8441/edge/client/v1"],"path":"/edge/client/v1"}},"edge-management":{"v1":{"apiBaseUrls":["https://localhost:18441/edge/management/v1"],"path":"/edge/management/v1"}}},"buildDate":"2022-10-13T15:31:04Z","revision":"72978b5aa932","runtimeVersion":"go1.19.1","version":"v0.26.10"},"meta":{}}

notice how the curl statement succeeds for you. So that explains why this is happening only on this machine. I’ll add a bug to the ziti edge enroll function to figure out a better way to handle this particular situation.

Until then - ‘fix/remove’ the edge controller’s CA from the OS and it will work as expected again.

bug filed identity is not usable when the ctrl/edge PKI are not the same · Issue #331 · openziti/sdk-golang · GitHub

1 Like
31

Ahh… I do remember doing that for some reason.

Tricky one to uncover… thanks so much for working it through :slight_smile:

I will definitely need to read up more on those certificates now

32

As a follow up to help my understanding.. it would be great if you can explain this in terms of what is happening between the OS trust store and the enrolment process..

When the CA is installed in the OS trust store.. does it bypass some test to determine where to source the CA bundle.. or something like that?

33

I filed a bug for this. It’s a flaw in how enrollment is working. identity is not usable when the ctrl/edge PKI are not the same · Issue #331 · openziti/sdk-golang · GitHub

But, it goes like this. When enrollment happens it makes a request to the controller API to decided if the ziti cli needs to pull the CA bundle, in order to successfully enroll. In your case - the first request succeeded (didn’t fail) due to the controller’s cert being considered trusted. Thus - there’s no need for the “ca” field to be populated (that’s the bug).

Now when your ziti-edge-tunnel (or whatever) would go to connect to the controller, things work fine, BUT when connecting to the edge router to establish the data plane, the quickstart uses a different PKI. Because of that, the router PKI is NOT trusted, and that leads the ziti-edge-tunnel to fail to connect.

If the quickstart used the same root CA, we’d never have seen this bug (that’s one of the reasons why the quickstart does it the way it does).

That enough details to make sense? If not, lemme know

1 Like
34

Ahh… makes sense now… thanks heaps for filling in the gaps. :slight_smile: