Ziti and spiffe

I was (re) watching the ziti tv episode about ziti and spiffe, and realized something which I didn’t before. @andrew.martinez mentioned that identity enrollment was not needed and tunneler can just use the identity.json file with cert and key details.

In my environment an identity is represented by a service ( not to be confused with ziti service ) and a given service can be bootstrapped on multiple hosts. So can I register a service once as an identity in ziti and irrespective of whether that is coming from host1 or host100, ziti would treat it as the same identity, given that identity name is mapped to [commonName] coming from the identity x509 certificate.

I feel this can work but would like to get expert opinion. TIA

1 Like

This is the workflow in my mind -

  1. Internal CA registration happens with [commonName] mapped to “identity-name-format” with ZIti controller
  2. Admin creates an identity using “ziti edge create identity service --external-id myservice” ( one time per identity )
  3. Through external to ziti workflow, 2 hosts are bootstrapped with an identity of myservice ( CN=myservice in the certificate issued by internal CA )
  4. ziti-edge-tunnel runs on both the hosts with identity.json pointing to internal CA issued cert and key and both hosts can talk to ziti services based on policies defined using the identity of “myservice”

Does this look correct?
Is there a need to have one time enrollment of the “myservice” identity using controller provided jwt or by using “–external-id” option, its not needed?

1 Like

@TheLumberjack Any suggestions on the above? ( In case you missed this question :slight_smile: )

Sorry, I saw that comment was on the “spiffe” topic and admittedly, I didn’t read the message since I’m not an expert there. However, I just read your message now and it doesn’t seem spiffe-related. My bad. If this is specifically spiffe related, I’m probably not much help there as I haven’t spent any time really with it.

Your flow does look correct, yes, but I’m not sure of the behavior of the system. I’ve been trying my own testing and trying to make some steps to illustrate “the steps” necessary, but I’ve not been able to get it quite right. I wanted to test what would happen if the same identity was used by two different instances like the way you’re proposing since I didn’t know if that would work or not.

My inclination is that the first identity would work properly, whereas the second one would not. That’s because the two hosts will have certificates that look the same, but the fingerprint of the certificate would be different. I expect ziti is going to store the fingerprint to the identity, so one would be valid and one won’t be, but I’m not enough of an expert on exactly how the edge controller would handle that situation. I expect @andrew.martinez would be able to verify that quicker than I can but I don’t think he’ll be available this week.

1 Like

hmm… I created the identity on the controller using

ziti edge create identity service myservice --external-id myservice -a 'myservice,client'

and then tried to start the ziti-edge-tunnel without enrollment and got the following error -

[        0.130]   ERROR ziti-sdk:ziti_ctrl.c:250 ctrl_login_cb() ctrl[mycontroller.example.com] INVALID_AUTH(The authentication request failed)
[        0.130]    WARN ziti-sdk:ziti.c:1391 api_session_cb() ztx[0] failed to get api session from ctrl[https://mycontroller.example.com:443/] api_session_state[1] INVALID_AUTH[-14] The authentication request failed
[        0.130]   ERROR ziti-sdk:ziti.c:1405 api_session_cb() ztx[0] identity[/opt/openziti/etc/identities/myservice.json] cannot authenticate with ctrl[https://mycontroller.example.com:443/]
[        0.130]    WARN tunnel-cbs:ziti_tunnel_ctrl.c:739 on_ziti_event() ziti_ctx controller connections failed: Not Authorized
[        0.130]   ERROR ziti-edge-tunnel:ziti-edge-tunnel.c:1185 on_event() ztx[/opt/openziti/etc/identities/myservice.json] failed to connect to controller due to Not Authorized

Does this enrollment-optional flow work only if the root CA has SPIFFE SAN URI?

Yeah that’s the sort of thing I was getting as well. It isn’t limited to spiffe only. We had auto enrollment in place for years, but to be honest I’ve not tried it out for a while, I expect I am not doing something quite right, but haven’t discovered yet exactly what I was doing wrong, yet. I might re watch the ziti tv video too for clues.

I’m going to give it an hour or two more today, but if I’m not successful we will probably have to wait until I can chat with @andrew.martinez. I’ll post the exact steps in using back here at the end of my own experiments.

1 Like

Thank you. :slightly_smiling_face:

I can make it work if instead of doing “ziti edge create identity …” if I run “ziti edge enroll … “ from the host and then the identity is created on the controller and auto enrolled. But I am hoping to the one time create and starting ziti-edge-tunnel with hand-crafted identity.json from multiple hosts would work because that will solve a lot of operational overhead.
Will wait until Andrew gets a chance to review this.

I am gonna write this up now. I forgot a step in my local testing, which I figured out today… I’ll post back in just a bit here.

1 Like

Ok. The short story… The step I kept forgetting about was the enrollment. That still needs to happen, even with an automatic identity. The difference is that the certificate itself will allow the enrollment/creation of the identity.

So here are the steps I performed…

Please note 1: I used an environment which was provisioned by the quickstart. That means I have a couple of environment variables in my environment. You will need to replace those accordingly. :slight_smile:

Please note 2: The ziti-edge-tunnel and ziti CLI supports 3rd party enrollment but the Ziti Desktop Edge for Mac/Windows don’t support that natively/easily yet. I don’t expect that’s an issue for you but, FYI. Also note that it seems that ziti-edge-tunnel has a bug for this enrollment as well that needs to be sorted. I had to use ziti to enroll the identities due to that apparent bug.

Automatic CA creation steps, leveraging ziti CLI

CA Name

establish a variable name of the CA we’ll be using. This variable will be used below, often


Create the x509 CA

This process will use the ziti CLI to create a certificate that is a CA. The product of these commands will also, by necessity, create a key. If you already have a key, you cannot currently provide it to the ziti CLI, unlike other pki commands. You’ll have to use openssl for now if you’re trying to do that.

ziti pki create ca \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-file="${ca_name}" \
  --pki-country "US" \
  --pki-locality "locality" \
  --pki-organization "org" \
  --pki-organizational-unit "the_ou" \
  --pki-province "ST" 

Add the CA to Your Overlay

Now you’ll need to send the certificate to the controller and register it as a third party cert. This command also demonstrates how to set the identity-format for the identities that will be automatically created, as well as setting the CA valid for auth, and autoca. Here’s another env var you’ll need to replace: ZITI_PKI_OS_SPECIFIC

ziti edge create ca "${ca_name}" \
  "${ZITI_PKI_OS_SPECIFIC}/${ca_name}/certs/${ca_name}.cert" \
  --auth \
  --autoca \
  --identity-name-format '[caName]-[caId]-[commonName]-[requestedName]-[identityId]'

Verify the CA

The CA needs to be verified by providing a certificate back to the controller that was signed by the CA you’re trying to use. The certificate must have the expected verification token in the certificate’s “CN” field.

For this process first get the verification token using the ziti CLI and jq and set it into the verification_token variable:

verification_token=$(ziti edge list cas 'name contains "'"${ca_name}"'"' -j \
  | jq -r .data[].verificationToken)

Now you can use the ziti CLI to create a certificate that can verify the CA, proving you have the key for the CA:

ziti pki create client \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name="${ca_name}" \
  --client-name="${verification_token}" \

Now we’ll actually verify the certificate. These steps will just located the verification cert and the ca cert/key for use in the next step:

verification_cert=$(find $ZITI_PKI_OS_SPECIFIC \
  -name "*${verification_token}*.cert")
cacert=$(find $ZITI_PKI_OS_SPECIFIC \
  -name "*${ca_name}*.cert")
cakey=$(find $ZITI_PKI_OS_SPECIFIC \
  -name "*${ca_name}*.key")

Let’s list the CA now and see the flags returned. Notice that the CA is not verified yet: flags: [AE]

ziti edge list cas 'name contains "'"${ca_name}"'"'

Now we can verify the CA

ziti edge verify ca "${ca_name}" \
  --cert "${verification_cert}" \
  --cacert "${cacert}" \
  --cakey "${cakey}"

And after being verified, we can list the CA and see it verified: flags: [VAE]

ziti edge list cas 'name contains "'"${ca_name}"'"'

At this point the CA is ready to be used.

The work to illustrate two clients

Now we’ll want to make two new sets of certificates which are signed by the CA made above, using ziti CLI again. If you have a private key, you could supply it here with --key-file.

ziti pki create client \
  --pki-root=${ZITI_PKI_OS_SPECIFIC} \
  --ca-name=${ca_name} \
  --client-name="${client1_name}" \
ziti pki create client \
  --pki-root=${ZITI_PKI_OS_SPECIFIC} \
  --ca-name=${ca_name} \
  --client-name="${client2_name}" \

Once created, we can set a couple of variables that represent the location of the client key and cert created above.

client1_cert="$(find $ZITI_PKI_OS_SPECIFIC \
  -name "*${client1_name}-file.cert")"
client1_key="$(find $ZITI_PKI_OS_SPECIFIC \
  -name "*${client1_name}-file.key")"

client2_cert="$(find $ZITI_PKI_OS_SPECIFIC \
  -name "*${client2_name}-file.cert")"
client2_key="$(find $ZITI_PKI_OS_SPECIFIC \
  -name "*${client2_name}-file.key")"

Now, we’ll need to fetch the jwt that an automatic, enrollment, 3rd party CA supports. Right now there’s no “really easy” way to fetch this CA so when you deploy a client, you’ll need to include this jwt as part of the provisioning step. The jwt is always available to download and only needs to be downloaded one time, however when you enroll an identity with ziti it’ll consume the .jwt. Just notice that, because below you’ll see me download the jwt two times.

First, we’ll need the session cookie returned when logging into the controller. Again, I’ve used a quickstart provisioned overlay, which means my $ZITI_HOME is set to $HOME/.ziti/quickstart/$(hostname).

Assign the use the output of ziti login to find your ziti-cli.json file and assign it to ziti_cli_json:


Use jq to get and set the zt_session from that file

zt_session=$(jq -r .edgeIdentities.default.token "${ziti_cli_json}")

Now get the id of the CA using ziti CLI and assign it to ca_id

ca_id=$(ziti edge list cas 'name = "'"$ca_name"'"' -j | jq -r .data[].id)

And we’re almost done… Now we can now list identities before enrollment… and then after…

ziti edge list identities 'name contains "sep08"'
curl -sk -H "zt-session: ${zt_session}" "https://${ZITI_EDGE_CTRL_ADVERTISED}/edge/management/v1/cas/${ca_id}/jwt" > "${ca_name}.jwt"
echo "fetched jwt: ${ca_name}.jwt" 
ziti edge enroll \
    --cert ${client1_cert} \
    --key ${client1_key} \
    --jwt ${ca_name}.jwt \
    --idname "${client1_name}.requestedName" \
    --out $HOME/${client1_name}.enrolled.json
curl -sk -H "zt-session: ${zt_session}" "https://${ZITI_EDGE_CTRL_ADVERTISED}/edge/management/v1/cas/${ca_id}/jwt" > "${ca_name}.jwt"
echo "fetched jwt: ${ca_name}.jwt"
ziti edge enroll \
    --cert ${client2_cert} \
    --key ${client2_key} \
    --jwt ${ca_name}.jwt \
    --idname "${client2_name}.requestedName" \
    --out $HOME/${client2_name}.enrolled.json

ziti edge list identities 'name contains "sep08"'

Using the identity file

./ziti-edge-tunnel run-host -i $HOME/${client1_name}.enrolled.json
[        0.099]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:801 on_ziti_event() ztx[sep08ca-vj6x99YBDI-sep08_one-sep08_one.requestedName-F-o7RnepDI] router ip-172-31-42-64-edge-router@tls://ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8442 connected

./ziti-edge-tunnel run-host -i $HOME/${client2_name}.enrolled.json
[        0.098]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:801 on_ziti_event() ztx[sep08ca-vj6x99YBDI-sep08_two-sep08_two.requestedName-Pi77RneBD] router ip-172-31-42-64-edge-router@tls://ec2-18-188-201-183.us-east-2.compute.amazonaws.com:8442 connected

Ok - and “that’s it”… :slight_smile: I know it looks like a lot but really it’s not all that complicated.

1 Like

this is great, although what I am really after is to use certificates with same CN from 2 diff hosts to be treated as a single identity by the controller.

Also did I interpret it incorrectly or @andrew.martinez mentioned in the YT video that enrollment is not needed? I am wondering if that is something only coming into play when CA has SPIFFE SAN URI.

If the CN important to you, can you confirm the identity name isn’t? I think you’ll be able to do that as long as you make two different identities via the “–idname”. Know what I mean? I imagine there’s a reason that you want to ‘aggregate’ identities like this, but I’m not seeing it yet.

oooh nevermind - I see that I’ve also lost track of the original question now. I see you explained it back there… I’ll try to carve out some more time to try to rewatch that video, and revisit the very original topic on this thread.

1 Like

Thank you :slightly_smiling_face: I have done pretty much everything you have listed here and these steps are very helpful for reference.

Alright, I think I get what you’re after now. Let me check my understanding. You’re looking to have one identity, with one external-id, then mint n separate certificates with that external-id in the cert so that any certificate with that external-id appears as that one singular identity. I think I got it.

Since this is so new, and I’ve not been able to get it to work in my own lab, we’re gonna have to wait for andrew to come back next week on this one. :slight_smile: We’ll pick it up in a few days.

1 Like

@TheLumberjack did you get a chance to check on this? If this can work as I am imagining it, it will be a huge help.

Alright. I got a chance to catch up with Andrew. Explained things and he says that if you are using external id’s, you’ll be able to have ‘n’ different key/cert pairs with the same external id show up and connect as that one identity successfully! That’s cool.

When creating the CA, you need to set the useExternalId field to true (a step I had missed when doing my own testing and in the steps above) and you would indicate the claimsProperty to be used as well.

So the answer is yep, that should work. I’m gonna try it out myself when I get a moment.

1 Like

Awesome! This made my day. :slight_smile:

Is it possible for me to edit this after-the-fact ( I already have a third party CA registered and verified ) I tried but couldnt find a way to add externalId and x509-claim to use for a given CA.

I guess I will wait for you to try and share wisdom!

:confused: looking at ziti CLI, it doesn’t seem to have useExternalId as part of the CLI yet. That’s a bummer. You’ll have to use the REST API directly through curl or something. I actually showed you how I use curl with the REST API in my big post from 6d ago. I don’t think the changelog has the field documented. Looking at the api docs (found at /edge/management/v1/docs#tag/Certificate-Authority/operation/createCa) the field name seems to be externalIdClaim which isn’t quite the same as useExternalId and useExternalId is on a different resource: /external-jwt-signers. It appears in the PUT and PATCH operations too, so it stands to reason that you’d be able to PATCH the field.

I won’t get around to try this myself for a while, probably next week. I’ll direct andrew to this thread tomorrow and convince him to get this thread sorted. :slight_smile:

But, we do know it’ll work once we get it figured out. heh

1 Like

I tried to run following command to patch but it didn’t work. Am I missing something?

curl -X PATCH "https://${CONTROLLER}/edge/management/v1/cas/${ca_id}" \
-H "Content-Type: application/json" -H "zt-session: ${zt_session}" \
-d '{"externalIdClaim":{"index":0,"location":"COMMON_NAME","matcher":"ALL","matcherCriteria":null,"parser":"NONE","parserCriteria":null}}'

Do you have the error output?

matcherCriteria and parserCriteria are required non-null values. They should be empty strings