Certificate Expired

OK - so good news - I have used Ziti for 1 year. Bad news, certificates expired, and now it is on the floor in a heap. This was originally configured, 1 year ago configured using the docker compose configuration. Connecting to the ziticontroller on 6262 gives me this certificate:

This is probably what this post was about to validate what was going to happen: Certificate life cycle - openziti

Anyhow, I cannot zitiLogin because the cert is expired. Should have these certificates auto-renewed, and therefore renewed the client side certificates at all?

What is the best way to resolve this…ie did I not have a configuration option selected.

Thoughts

1 Like

Looking at the signing CA, the ziti-signing-root-ca.cert has a

notAfter=Jun 2 20:43:09 2032 GMT

so if it was going to regenerate identities, it is not that it couldn’t because the CA expired.

1 Like

Just to clarify, it was the quickstart containers that I was using. Also, the identity that pointed to this controller had a long name: b508c6c1b9dca586f46ce88a7520c23d6d64e13a.json. I doubt that I would have had it as this. Checking the expiry time did not show a certificate renewal occured. Certificate expires near the same time.

For certs ziti is managing, it sure seems reasonable that it they’d renew but that either didn’t happen or there’s a big preventing it from succeeding. I will see if I can come up with a short set of steps that will get you back running. I would think this is only a matter of generating a new certificate for your controller using the key created during the quickstart.

I expect your identities are failing to connect to the controller and you can’t use the ziti cli. Can you confirm some kind of pki related error in the client logs?

I would think this will be one ziti pki command, followed by a config update and controller restart. I’ll get cracking on it

1 Like

UPDATED AUG-20-2024. If these commands are not working it's because the environment variables have changed. Please see this post: Certificate renewal showing path //keys/-server.key - #3 by TheLumberjack and then come back here after ensuring the env vars are set properly


Alright. I think I have the set of steps that will fix your server certificates. I believe you should be able to replace just the server certs. Since it's been a year, I hope the quickstart variables haven't changed too much and all these commands work.

Establish some variables and make sure they look right

First, setup some vars. It's important the dns/ip variables are full and complete are are what you expect them to be. Make sure you have set the EXTERNAL_DNS and ZITI_EDGE_CONTROLLER_IP_OVERRIDE if you have set them. Also make sure you have the .env file sourced (or all the referenced variables shown below set properly)

Run all these commands and at the end it'll echo the values it found. verify those values. Here's my example:

export ZITI_CTRL_IDENTITY_KEY="${ZITI_PKI_OS_SPECIFIC}/${ZITI_CONTROLLER_INTERMEDIATE_NAME}/keys/${ZITI_CONTROLLER_HOSTNAME}-server.key"
export ZITI_EDGE_CTRL_IDENTITY_KEY=export ZITI_EDGE_CTRL_IDENTITY_KEY="${ZITI_PKI_OS_SPECIFIC}/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}/keys/${ZITI_EDGE_CONTROLLER_HOSTNAME}-server.key"
now="$(date '+%Y-%m-%d-%M%S')"

pki_allow_list_dns="${ZITI_CONTROLLER_HOSTNAME},localhost,${ZITI_NETWORK}"
if [[ "${ZITI_EDGE_CONTROLLER_HOSTNAME}" != "" ]]; then pki_allow_list_dns="${pki_allow_list_dns},${ZITI_EDGE_CONTROLLER_HOSTNAME}"; fi
if [[ "${EXTERNAL_DNS}" != "" ]]; then pki_allow_list_dns="${pki_allow_list_dns},${EXTERNAL_DNS}"; fi
pki_allow_list_ip="127.0.0.1"
if [[ "${ZITI_EDGE_CONTROLLER_IP_OVERRIDE}" != "" ]]; then pki_allow_list_ip="${pki_allow_list_ip},${ZITI_EDGE_CONTROLLER_IP_OVERRIDE}"; fi

echo "DNS ENTRIES  : $pki_allow_list_dns"
echo "IP ENTRIES   : $pki_allow_list_ip"
echo "CTRL KEY FILE: ${ZITI_CTRL_IDENTITY_KEY}"
echo "EDGE KEY FILE: ${ZITI_EDGE_CTRL_IDENTITY_KEY}"

My example output:

DNS ENTRIES  : ip-172-31-47-200,localhost,ip-172-31-47-200,ec2-3-134-108-218.us-east-2.compute.amazonaws.com,ec2-3-134-108-218.us-east-2.compute.amazonaws.com
IP ENTRIES   : 127.0.0.1,3.134.108.218
CTRL KEY FILE: /home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/keys/ip-172-31-47-200-server.key
EDGE KEY FILE: /home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ec2-3-134-108-218.us-east-2.compute.amazonaws.com-intermediate/keys/ec2-3-134-108-218.us-east-2.compute.amazonaws.com-server.key

Create new server certs

Now you need to make server certs for the controller api (the overlay network) and for the edge. NOTICE I have added the "expire-limit" flag on my example. I had patched the ziti-cli and set my limit to 'minutes' to actually test this. Hopefully, I did a good and thorough job testing (I tried to).

Make the cert for the overlay components:

"${ZITI_BIN_DIR-}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file ${ZITI_CONTROLLER_HOSTNAME}-server \
  --server-file "${ZITI_CONTROLLER_HOSTNAME}-server.${now}" \
  --dns "${pki_allow_list_dns}" \
  --ip "${pki_allow_list_ip}" \
  --expire-limit 1200 \
  --server-name "${ZITI_CONTROLLER_HOSTNAME} server certificate ${now}"

Make the cert for the edge:

"${ZITI_BIN_DIR-}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file ${ZITI_EDGE_CONTROLLER_HOSTNAME}-server \
  --server-file "${ZITI_EDGE_CONTROLLER_HOSTNAME}-server.${now}" \
  --dns "${pki_allow_list_dns}" \
  --ip "${pki_allow_list_ip}" \
  --expire-limit 1200 \
  --server-name "${ZITI_EDGE_CONTROLLER_HOSTNAME} server certificate ${now}"

Update the controller config file

Now find the files created using:

find $ZITI_PKI_OS_SPECIFIC -name '*'${ZITI_CONTROLLER_HOSTNAME}'*'${now}'*.chain.pem'
find $ZITI_PKI_OS_SPECIFIC -name '*'${ZITI_EDGE_CONTROLLER_HOSTNAME}'*'${now}'*.chain.pem'

The first file result returned from the commands is the cert for the overlay components. Find the identity.server_cert section and update it with this value.

The second result is the edge cert to use. Find it in the web section under probably the name "client-management".

Here's a video of a walkthrough of regenerating the server certs using the ziti cli:

1 Like

Thanks for the steps. Yes I am up and running again. Client auto-renewed certificates as well which is a relief (or are at least working)!
For me, probably old compose file, but the my configuration has the controller and edge router using the same certificate. Great that you have detailed separately. For the quick start, everything is on the one host.
For the record, I have to update the certs in two places in the /persistent/ziti-edge-controller.yaml.

What is interesting, is that the edgerouter certificates appear to have updated the server.cert and client.cert on May 29th (~ 10 days before cert expired). This did not happen on the controller, but does not seem to be using them (see below table)

All seems to be up and going, although I might be in for some more trouble

ziticontroller.thesmithcave.nz:1280 = expires 24
ziticontroller.thesmithcave.nz:6262 = expires 24
ziticontroller.thesmithcave.nz:3022 = expired

even though I can still connect through that gateway and showing as online. Not sure that should be correct but is working at the moment.

So, should certificates be auto-renewing? I dont have logs as lost when restarting docker. Is this likely a bug.

As you saw, routers definitely should auto renew. I don't think the controller was ever setup to do it, so it's probably more a feature request for the controller than a bug per-se, but it seems like a nice/friendly thing to do. With HA coming, this might already be accounted for, I'll mention it to the team and see what the consensus is.

1 Like

Thanks @TheLumberjack. I might log an issue ticket. Not sure if you are still doing monthly get togethers, but definitely something that I think needs to be included. Especially as there is no easy way of getting notified of upcoming outage.

Done: Feature Request: Auto renewal of controller (and any other) certificates · Issue #1155 · openziti/ziti · GitHub

We’re coming up on a year for some ziti clusters and cert rotation and potential downtime is a concern for us also. So this info is a big help, thanks!

With this thread, we have instructions to re-create / rotate required controller certs that were generated in the quickstart with a relatively short term default (1 yr).

The router cert doesn’t look like an issue because as mentioned above it auto-updates as long as controller certs are valid.

For ziti clients (ex: deployed ziti-edge-tunnelers) which had JWT credentials manually generated from ziti-admin-console (ZAC), the resulting json identity files look like the embedded certs also default to a year, and from what I understand, they will require manual renewal/rotation (please correct me if I’m wrong) to avoid downtime at the expiration date.

I’m not using the latest ZAC but instead from a commit a few months ago. I’ll have to upgrade soon. If this isn’t already included in ZAC, here’s an idea: show a column in the “Manage identity” window for identity expiration. Right now there is a “Created” column with the created date, so I suppose I can just add a year to everything to know the expiration if that assumption is valid.

Another [maybe better] idea would be to have a dedicated page in ZAC for all ziti related certs with expiration that an admin can check at a glance to know what type of required actions might be coming up for cert rotation.

That would be easier for most compared to the poking around that may otherwise need to be done to discover which expiration problems are about to happen.

I need to verify this but I am 90% sure that by default, ziti does NOT enforce certificate expiration. I'm also pretty sure if you want to, you can change openziti to respect the certificate expiration but as you mention that can possibly be a problem, so it's not enabled by default.

I appreciate the ZAC ideas. I'll make sure the team sees it and we'll see what we can do around this.

It'll probably be a few days before I can confirm the above, but I'm pretty sure that's how it works.

1 Like

The certificates for the clients will continue to work, even after the year expiry. That is what I am seeing.

1 Like

Thanks for the responses! Sounds like we may be good all around then in preparing for cert rotations and likely not needing to rotate the manually set up clients if we don’t want to. If some caveat comes up we still have at least a few weeks before anything starts expiring, so plenty of time still.

Happy July 1st / July 4th, etc, to those having a holiday weekend!

Been thinking about this, and put in a feature request (Feature Request: Banner for certificate expiry · Issue #148 · openziti/ziti-console · GitHub) to help notify people. Currently logged under ZAC, but would want that for the console as well.

Picking up on this. We just had the issue of cert validity running out.
The server renewal worked as described, thanks @TheLumberjack. However when viewing the client cert
openssl x509 -in /home/ziti/.ziti/quickstart/zt/pki/zt-intermediate/certs/zt-client.cert -text, I can see that it didn't renew:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            8e:7c:8c:e5:30:d2:fb:90:2b:bf:b3:bf:34:bb:22:1a
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, L = Charlotte, O = NetFoundry, OU = ADV-DEV, CN = zt-intermediate
        Validity
            Not Before: Oct 15 22:24:57 2022 GMT
            Not After : Oct 15 22:25:57 2023 GMT

The edge certificate did successfully renew on October 9th it seems.
How should we proceed with the client cert?

@dmuensterer yes, go ahead and manually update the client cert. It seems either we have some intermittent behavior or something changed along the way as, in my case, both of my client certs had expired. The command would look something like this (following along variables used in the example above.

"${ZITI_BIN_DIR-}/ziti" pki create client \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file "${ZITI_CONTROLLER_HOSTNAME}-server.${now}" \
  --expire-limit 1200 \
  --client-name "${ZITI_CONTROLLER_HOSTNAME} client certificate ${now}"
1 Like

Thanks. For the identities, do I need to renew certificates here? Can I check the validity? How does the renewal process work?

Thanks!

@dmuensterer currently, identities never renew so there is no need to worry about those, they should be good to go.

I’m not sure I fully understand.
There's an x509 certificate for every enrolled identity... but it's not checked for expiration validity? Is that what you mean?

Thanks.

Using the defaults that ship with an OpenZiti controller, yes that's correct. The default Auth policy is configured to allow expired certificates. You can change this behavior but there are very valid reasons to allow an edge identity to be permitted to authenticate regardless of the certificates values. Hth

1 Like

It does, thanks! Means by default not to worry about expiring client certs!

1 Like