Creating Endpoint with your own Certificate Authority

Quick update… as I am still stuck on this… maybe I am doing something wrong as I created the Certificate Authority manually… and then imported.

When I did some more sleuthing… and I came across this interesting script… that uses the ziti pki commands

After adjusting this for what I needed… here are the commands that I used.

ziti edge list cas

You can also see that it has been verified

read -s adminpwd




zt_session=$(curl -sk -H “Content-Type: application/json”
-d “{“username”:“admin”,“password”:”${adminpwd}"}" |
jq -r .data.token)

the_ca=$(curl -X GET -sk -H “Content-Type: application/json” -H “zt-session: ${zt_session}” “${edge_controller_uri}/cas/${ca_id}” | jq -r “.data”)

echo $the_ca

the result is null

echo $zt_session

so for some reason… so I did a bit more walkabout

curl ${edge_controller_uri}/cas/${ca_id}

curl: (60) SSL certificate problem: unable to get local issuer certificate

More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not

establish a secure connection to it. To learn more about this situation and

how to fix it, please visit the web page mentioned above.

Any tips?

just an fyi… I will be getting back to this “tomorrow” (I’m pretty sure it’s already “tomorrow” for you :slight_smile: ) I haven’t forgotten you!

All ok… it gave me time to really understand this… and explore lots of other options… twists and turns… and other new surprises.

I am looking forward to working through this… as it will help me unlock my next mastery level :slight_smile:

The main stumbling block is downloading the jwt file… as there is something not quite right with my approach

I almost sent you those scripts last week but didn’t have time to describe them nor to test them. I was working up an example based on those for you. I’m glad you discovered them but I’m also glad that I tested them out since they are slightly wrong now. :slight_smile: Not long ago we did an ‘api split’ which breaks these older instructions.

I have gone about repairing the instructions and they “mostly” work but not fully. I am unsure if there’s a bug or if I’ve done it wrong. I will have to dig in a bit more. I’ll post back after I’ve had a chance to debug more.

1 Like

I actually was able to figure it out after a tiny bit of scratching around. This will be a big post. Let me put it together and I’ll reply in a few

Ok. The video is below. The key is the name of the identity. It needed to follow the “[caName]-[commonName]” format or else it wouldn’t connect. Also the api paths were no longer valid. Good luck - hopefully this helps

Here are the full instruction set - it’s quite long and ‘dense’… [Edited - ZITI_PKI]

# generate a new CA
ca_name="new_ca_$(date +"%H%M%S")"

echo "New CA is named   : ${ca_name}"
echo "New CA directory  : ${new_ca_dir}"

# create the PKI using the CLI
ziti pki create ca \
    --pki-root="${ZITI_PKI}" \
    --ca-file "${ca_name}" \
    --ca-name "${ca_name}"


# use ZAC and add the CA
cat "$ZITI_PKI/${ca_name}/certs/${ca_name}.cert"


# use your env file to get a zt_session header - used for curls below
export zt_session=$(curl -sk -H "Content-Type: application/json" \
    https://${ZITI_EDGE_CTRL_ADVERTISED}/authenticate?method=password \
    -d "{\"username\":\"${ZITI_USER}\",\"password\":\"${ZITI_PWD}\"}" \
    | jq -j .data.token)

# grab the id of the ca you just created - used below
ca_id=$(curl -sk \
    -H "Content-Type: application/json" \
    -H "zt-session: ${zt_session}" \
    "https://${ZITI_EDGE_CTRL_ADVERTISED}/edge/management/v1/cas" \
    | jq -j '.data[].id')
echo "New CA resides at : $ZITI_PKI/${ca_name}/certs/${ca_name}.cert"
echo "New CA has id     : ${ca_id}"


# Using ZAC - find the verification id. verify the CA cert...
identity_to_verify="<copy verification id here>"
ziti pki create client \
    --pki-root="${ZITI_PKI}" \
    --ca-name=${ca_name} \
    --client-name=${identity_to_verify} \
echo "New client lives at   : $ZITI_PKI/${ca_name}/certs/${identity_to_verify}.cert"


# set your identity name. this is VITAL you need to use the format of "[caName]-[commonName]
# you can see when looking at the json that this is output:
#      "identityNameFormat": "[caName]-[commonName]",
# this was my missing step. presenting a cert that doesn't match this pattern makes it fail to auth      
identity_name="${ca_name}-ca_id_$(date +"%H%M%S")"

echo "New Identity named: ${identity_name}"

ziti pki create client \
    --pki-root="${ZITI_PKI}" \
    --ca-name=${ca_name} \
    --client-name=${identity_name} \

# create a new identity - I couldn't find a 'ziti cli' nor 'ZAC' way of doing this. Needed to use the API
identity_id=$(curl -sk \
    -H "Content-Type: application/json" \
    -H "zt-session: ${zt_session}" \
    "https://${ZITI_EDGE_CTRL_ADVERTISED}/edge/management/v1/identities" \
    -d '{ "name": "'"${identity_name}"'", "type": "User", "isAdmin":false, "enrollment": { "ottca": "'"${ca_id}"'" } }' \
    | jq -j ''

echo "Third Party OTT identity created. ID: ${identity_id}"

# get the jwt from the controller - used to enroll...
curl -sk -H "Content-Type: application/json" \
     -H "zt-session: ${zt_session}" \
     "https://${ZITI_EDGE_CTRL_ADVERTISED}/edge/management/v1/identities/${identity_id}" \
     | jq -j .data.enrollment.ottca.jwt > ${jwt_file}
echo "using jwt at ${jwt_file} to enroll"

# you need the CA bundle in order to enroll - this command grabs the ca bundle
curl -sk https://${ZITI_EDGE_CTRL_ADVERTISED}/.well-known/est/cacerts > ${ZITI_PKI}/fetched-ca-certs.p7
openssl base64 -d -in ${ZITI_PKI}/fetched-ca-certs.p7 | openssl pkcs7 -inform DER -outform PEM -print_certs -out ${ZITI_PKI}/fetched-ca-certs.pem

# actually enroll the identity
ziti edge enroll \
    --jwt "${jwt_file}" \
    --cert "$ZITI_PKI/$ca_name/certs/${identity_name}.cert" \
    --key "$ZITI_PKI/$ca_name/keys/${identity_name}.key" \
    --idname "${identity_name}" \
    --ca "${identity_full_ca_path}" \
    --out "$ZITI_PKI/$ca_name/keys/${identity_name}.json"

ziti edge create config '' host.v1 '{"protocol":"tcp", "address":"","port":80}'
ziti edge create config 'eth0.intercept.v1' intercept.v1 '{"protocols":["tcp"],"addresses":["eth0.discourse.ziti"], "portRanges":[{"low":80, "high":80}]}'
ziti edge create service 'eth0' --configs 'eth0.intercept.v1',''

ziti edge create service-policy 'eth0.binding' Bind --service-roles '@eth0' --identity-roles "@${identity_name}"

ziti edge create service-policy 'eth0.dialing' Dial --service-roles '@eth0' --identity-roles '@eth0.client'

# run the ziti-edge-tunnel with our newly provisioned identity
sudo ~/ziti-edge-tunnel run -i "$ZITI_PKI/$ca_name/keys/${identity_name}.json"

# delete the 'eth0' services/configs if needed
ziti edge delete config ''
ziti edge delete config 'eth0.intercept.v1'
ziti edge delete service 'eth0'
ziti edge delete service-policy 'eth0.binding'
ziti edge delete service-policy 'eth0.dialing'
1 Like

All ok… it has provided time to work through the difference between a self signed certificate authority… and a public certificate authority.

its like opposite ends of the string.

The reason why this is so important is because I need to setup a simple test environment to showcase a golang reverse proxy over a dark network using Ziti

If I can work through the process of “verifiying” a new self signed certificate authority… and create new identities from it… I should be able to get my demonstration working

The problem all started when I wanted to host a proxy server for an Apex application. The server automatically created a self signed certificate which worked a treat… until I tried to connect to the server using a golang reverse proxy.

The Ziti network does not recognise the self signed certificate because its not from a trusted source (ie. a public certificate authority).
This is also why the reverse proxy works for my website… because its created from a public certificate authority… with wildcard DNS and SAN

I did register a public DNS for the server to have a certificate created by a public certificate authority… this did solve the “trust” issue… but created other errors because the certificate did not have a wildcard DNS… nor did it use SAN… which are needed for the golang reverse proxy to work

Unfortunately… the free “certificate authorities” do not allow you to have wildcard DNS names or SANs very easily… and its not something I was able to setup with my current providers
I could use another register to create a certificate with a wildcard DNS and SAN but its a bit more than I was wanting to pay… as it’s only for a one time use purpose.

So… to do a bit more testing… I sourced the root certificate and key from the Ziti controller… thinking this was going to solve my problem… because it would allow me to create a server certificate that should be trusted by the controller.

this did not work

I failed big time after a lot of effort to set this up… because it stopped the server from starting up.

you cannot do this

The problem is… when you start a server over TLS… it will use the certificate and key to validate the origin of the server… so its going to go looking for that public certificate authority… but it cannot find it…

this is similar to the error you get when you open up a web page that secured with a self signed certificate…

What confused me a lot … was that the server can start with a self signed certificate authority… because… it is the certificate authority

however… when you visit your website over a browser… it does not recognised the certificate authority

this one difference is something stamped on my forehead now :slight_smile:

Hence, when you use a server certificate created from the ziti controller certificate authority…it cannot find the ziti controller self signed certificate authority … causing the server to fail to start.

what is the error message you get back
?? nothing,

All you observe is… sorry… the server did not start… it does not provide any explanation as to why…other than… a cryptic reference to your certificate key with no explanation as to why

So I am now back to the starting point… of using a self signed certificate authority in the format that I need

openssl req -new -sha256 -key server.key -out server.csr -subj “/"
-addext "subjectAltName=DNS:,”

I then need to verify this certificate authority in Ziti by creating a client certificate with the validation key…

Once this is done… I then need to create a new server identity for the reverse proxy that is created with the “trusted” certificate authority.

there are two specific problems that I am having with this.

#1. I cannot download the certificate authority JWT file to create auto enrolled identities
#2. I can create identities with the verified self signed certificate authority… but I cannot download the identity JWT file to enrol it.

These two specific points are where I am stuck

This was why I started to look around … to see if there was any reference point that I can learn from…

I look forward to your response… and keen to test it… as if I can work through to the point of creating an enrolled identity from a verified self signed certificate authority… it will be a massive milestone… and time for a small (large) celebration

Hopefully… my experience can help others also going through this experience… its a whole new world outside AppDev… that I never fully appreciated… but now I certainly do… its a minefield … ::slight_smile:

PS. just noticed your response… thanks… and will revert back for my result :slight_smile:

One question… is there a way to specify a SAN … I noticed that the example only makes reference to the commonName ?

My goal was to illustrate the process using ziti cli commands and illustrate “how to” with that. The CA authentication is verifying the format of the identity follows the expected format.

I’m not exactly sure what you’re after with the SAN field. Personally I’ve never branched out from the ziti cli because PKI is an incredibly complex topic and easy to get wrong, and easy to get confused/lost.

What are you looking for the SAN for? What purpose?

1 Like

Thanks also for the detail… btw… very helpful.

I will definitely work through this line by line… and validate that I don’t need a SAN.

this is one of the errors I received when using a certificate in my earlier tests

x509: certificate relies on legacy Common Name field, use SANs instead

Here is a reference article

If I get stuck… what I will try is to create the self signed certificate myself… then upload and verify… then use the commands to download the JWT.

I will keep you updated…

Quick update… as I still have a bit more to work through… but I can definitely say…

#1. I have verified the Certificate Authority using your instructions
#2. I have enrolled the client identity using your instructions

Thank you soooooo much… and a big high five @TheLumberjack … as the instructions were super helpful… without this… I would never have progressed.

a little bit of context… to get the specific outcome that I wanted… I needed to do a little bit of hacking around the edges
Specifically… I needed to create a CA with a specific common name with SAN… I could do the common name… but not the SAN using the pki command provided.

To create the Certificate Authority that I wanted with the SAN I needed… I created it separately with all of the specifications that I needed.

Once created… I just dropped in the new certificate and keys… then kept going

One point to mention… is that this approach does not support paraphrases… so if you are considering doing something like this… you need to remove the paraphrase from the private key

openssl rsa -in -out

A bit more troubleshooting insights
When I was using the certificate and key to start up a server running over SSL… I had an awful lot of trouble to get it to work…

Guess what the problem was?

File format… the contents of the private key and certificates were exactly the same… just stored in a different file format… despite it having the same extension.

Wow… so the one key learning here is… always cat the key and certificate to make sure this does not happen… as this is the specific trick that I did to get back on track with this.

Success… :slight_smile:

Here is the page from an Oracle Apex application hosted over a public DNS

Here is the same page being rendered from a goloang reverse proxy server over a ziti network…

What is special about this is the certificate… and the identity used to dial the service…

the certificate is self signed and “verified” by the Ziti controller…

And the identity used to dial the service is associated with the self signed certificate authority… as opposed to being associated with the Ziti controller root CA chain…

as @TheLumberjack highlights above… the name needs to be in a specific format… otherwise… its not going to work

For the reverse proxy to work… here is how I setup the certificate…

Without the subject alternative name… it will not work… and generate an error

x509: certificate relies on legacy Common Name field, use SANs instead

One thing I did need to do was to suppress the following message in golang

x509: certificate signed by unknown authority self signed certificate

this is because the server certificate was created from a self signed certificate…

If this was a production environment… I would need to remove this exception… and use a different certificate.

All up… the main learning was… certificates are tricky… and it is quite challenging to build a dev / test environment using self signed certificates… because things break if you do

Thanks again @TheLumberjack … for me… this was one of the hardest things I have ever done… I went down so many dead end roads…

Once you walked through the demo re creating a certificate authority… it all fell into place.

Now… its time to rest and celebrate … can’t believe it to be honest.


1 Like

Isn’t that THE TRUTH!!! :slight_smile: They sure can be finicky, and complex, particularly when first starting on that journey!

You could look into fixing the “x509: certificate signed by unknown authority self signed certificate” issue by discovering how to instruct golang to use a ca bundle you specify. :slight_smile: that is why in the steps above I have the steps included to pull the ca bundle down. Instead of quieting the warning, ziti will refuse to connect if the cert is unkown (a best practice)

So these steps:

# you need the CA bundle in order to enroll - this command grabs the ca bundle
curl -sk https://${ZITI_EDGE_CTRL_ADVERTISED}/.well-known/est/cacerts > ${ZITI_PKI}/fetched-ca-certs.p7
openssl base64 -d -in ${ZITI_PKI}/fetched-ca-certs.p7 | openssl pkcs7 -inform DER -outform PEM -print_certs -out ${ZITI_PKI}/fetched-ca-certs.pem

Those go fetch the ‘ca bundle’ from the controller and put them into a well known location. That CA bundle is then used in enrollment:

ziti edge enroll \
    **--ca "${identity_full_ca_path}"** HERE \

Nice job getting things working!

1 Like

Thanks so much… I was so excited that I got past the downloading of a JWT file… I must have missed that bit… I still cannot believe it.

As a quick additional comment… I was cleaning up all of my workings and took a look through the router yaml file.

What is interesting is a number of things

  1. You define the CSR
  2. You can include SANs

This may change how redo this test in the future… not sure at this point in time…


country: US
province: NC
locality: Charlotte
organization: NetFoundry
organizationalUnit: Ziti

# (required) SANs that this Gateways certs should contain. At least one IP or DNS SAN should be defined that matches
# the edge listeners "advertise" value from the "listeners" section.
    - "localhost"
    - "test-network"
    - "test-network.localhost"
    - "ziti-dev-ingress01"
    - ""
    - ""
    - "ziti://ziti-dev-gateway01/made/up/example"

I checked through the files and found the bundled ca certs file

Screen Shot 2022-05-04 at 10.19.44 am

Though… what do you do with this file… is this something you copy to the server with the private key?

Let me know if you have any tips

It’s used when enrolling. you MUST provide the full trust store when using openziti. you cannot bypass the x509 check. these commands pull the file, unzip, un “p7” it etc… and put it into that location. it’s then used when enrolling

# you need the CA bundle in order to enroll - this command grabs the ca bundle
curl -sk https://${ZITI_EDGE_CTRL_ADVERTISED}/.well-known/est/cacerts > ${ZITI_PKI}/fetched-ca-certs.p7
openssl base64 -d -in ${ZITI_PKI}/fetched-ca-certs.p7 \
    | openssl pkcs7 \
        -inform DER \
        -outform PEM \
        -print_certs \
        -out ${ZITI_PKI}/fetched-ca-certs.pem  <-- right here

reorganized to make it a bit more clear

Here is a screen shot of the enrolled identity… is there a way where I can check that I have it done correctly with the bundled ca cert?

I don’t know if there’s a clear way to tell. The best way would be to look at the identity file (the .json) and verify the cert that is used in that file was signed by the CA. You could also try to attach a ziti-edge-tunnel and verify it works to pass traffic.

1 Like