Certificate Expired

[gooseleggs]

OK - so good news - I have used Ziti for 1 year. Bad news, certificates expired, and now it is on the floor in a heap. This was originally configured, 1 year ago configured using the docker compose configuration. Connecting to the ziticontroller on 6262 gives me this certificate:

This is probably what this post was about to validate what was going to happen: Certificate life cycle - openziti

Anyhow, I cannot zitiLogin because the cert is expired. Should have these certificates auto-renewed, and therefore renewed the client side certificates at all?

What is the best way to resolve this…ie did I not have a configuration option selected.

Thoughts

[gooseleggs]

Looking at the signing CA, the ziti-signing-root-ca.cert has a

notAfter=Jun 2 20:43:09 2032 GMT

so if it was going to regenerate identities, it is not that it couldn’t because the CA expired.

[gooseleggs]

Just to clarify, it was the quickstart containers that I was using. Also, the identity that pointed to this controller had a long name: b508c6c1b9dca586f46ce88a7520c23d6d64e13a.json. I doubt that I would have had it as this. Checking the expiry time did not show a certificate renewal occured. Certificate expires near the same time.

[TheLumberjack]

For certs ziti is managing, it sure seems reasonable that it they’d renew but that either didn’t happen or there’s a big preventing it from succeeding. I will see if I can come up with a short set of steps that will get you back running. I would think this is only a matter of generating a new certificate for your controller using the key created during the quickstart.

I expect your identities are failing to connect to the controller and you can’t use the ziti cli. Can you confirm some kind of pki related error in the client logs?

I would think this will be one ziti pki command, followed by a config update and controller restart. I’ll get cracking on it

[TheLumberjack]

Alright. I think I have the set of steps that will fix your server certificates. I believe you should be able to replace just the server certs. Since it’s been a year, I hope the quickstart variables haven’t changed too much and all these commands work.

Establish some variables and make sure they look right

First, setup some vars. It’s important the dns/ip variables are full and complete are are what you expect them to be. Make sure you have set the EXTERNAL_DNS and ZITI_EDGE_CONTROLLER_IP_OVERRIDE if you have set them. Also make sure you have the .env file sourced (or all the referenced variables shown below set properly)

Run all these commands and at the end it’ll echo the values it found. verify those values. Here’s my example:

export ZITI_CTRL_IDENTITY_KEY="${ZITI_PKI_OS_SPECIFIC}/${ZITI_CONTROLLER_INTERMEDIATE_NAME}/keys/${ZITI_CONTROLLER_HOSTNAME}-server.key"
export ZITI_EDGE_CTRL_IDENTITY_KEY=export ZITI_EDGE_CTRL_IDENTITY_KEY="${ZITI_PKI_OS_SPECIFIC}/${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}/keys/${ZITI_EDGE_CONTROLLER_HOSTNAME}-server.key"
now="$(date '+%Y-%m-%d-%M%S')"

pki_allow_list_dns="${ZITI_CONTROLLER_HOSTNAME},localhost,${ZITI_NETWORK}"
if [[ "${ZITI_EDGE_CONTROLLER_HOSTNAME}" != "" ]]; then pki_allow_list_dns="${pki_allow_list_dns},${ZITI_EDGE_CONTROLLER_HOSTNAME}"; fi
if [[ "${EXTERNAL_DNS}" != "" ]]; then pki_allow_list_dns="${pki_allow_list_dns},${EXTERNAL_DNS}"; fi
pki_allow_list_ip="127.0.0.1"
if [[ "${ZITI_EDGE_CONTROLLER_IP_OVERRIDE}" != "" ]]; then pki_allow_list_ip="${pki_allow_list_ip},${ZITI_EDGE_CONTROLLER_IP_OVERRIDE}"; fi

echo "DNS ENTRIES  : $pki_allow_list_dns"
echo "IP ENTRIES   : $pki_allow_list_ip"
echo "CTRL KEY FILE: ${ZITI_CTRL_IDENTITY_KEY}"
echo "EDGE KEY FILE: ${ZITI_EDGE_CTRL_IDENTITY_KEY}"

My example output:

DNS ENTRIES  : ip-172-31-47-200,localhost,ip-172-31-47-200,ec2-3-134-108-218.us-east-2.compute.amazonaws.com,ec2-3-134-108-218.us-east-2.compute.amazonaws.com
IP ENTRIES   : 127.0.0.1,3.134.108.218
CTRL KEY FILE: /home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ip-172-31-47-200-intermediate/keys/ip-172-31-47-200-server.key
EDGE KEY FILE: /home/ubuntu/.ziti/quickstart/ip-172-31-47-200/pki/ec2-3-134-108-218.us-east-2.compute.amazonaws.com-intermediate/keys/ec2-3-134-108-218.us-east-2.compute.amazonaws.com-server.key

Create new server certs

Now you need to make server certs for the controller api (the overlay network) and for the edge. NOTICE I have added the “expire-limit” flag on my example. I had patched the ziti-cli and set my limit to ‘minutes’ to actually test this. Hopefully, I did a good and thorough job testing (I tried to).

Make the cert for the overlay components:

"${ZITI_BIN_DIR-}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file ${ZITI_CONTROLLER_HOSTNAME}-server \
  --server-file "${ZITI_CONTROLLER_HOSTNAME}-server.${now}" \
  --dns "${pki_allow_list_dns}" \
  --ip "${pki_allow_list_ip}" \
  --expire-limit 1200 \
  --server-name "${ZITI_CONTROLLER_HOSTNAME} server certificate ${now}"

Make the cert for the edge:

"${ZITI_BIN_DIR-}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name "${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME}" \
  --key-file ${ZITI_EDGE_CONTROLLER_HOSTNAME}-server \
  --server-file "${ZITI_EDGE_CONTROLLER_HOSTNAME}-server.${now}" \
  --dns "${pki_allow_list_dns}" \
  --ip "${pki_allow_list_ip}" \
  --expire-limit 1200 \
  --server-name "${ZITI_EDGE_CONTROLLER_HOSTNAME} server certificate ${now}"

Update the controller config file

Now find the files created using:

find $ZITI_PKI_OS_SPECIFIC -name '*'${ZITI_CONTROLLER_HOSTNAME}'*'${now}'*.chain.pem'
find $ZITI_PKI_OS_SPECIFIC -name '*'${ZITI_EDGE_CONTROLLER_HOSTNAME}'*'${now}'*.chain.pem'

The first file result returned from the commands is the cert for the overlay components. Find the identity.server_cert section and update it with this value.

The second result is the edge cert to use. Find it in the web section under probably the name “client-management”.

Here’s a video of a walkthrough of regenerating the server certs using the ziti cli:

Renewing certs that have expired

[gooseleggs]

Thanks for the steps. Yes I am up and running again. Client auto-renewed certificates as well which is a relief (or are at least working)!
For me, probably old compose file, but the my configuration has the controller and edge router using the same certificate. Great that you have detailed separately. For the quick start, everything is on the one host.
For the record, I have to update the certs in two places in the /persistent/ziti-edge-controller.yaml.

What is interesting, is that the edgerouter certificates appear to have updated the server.cert and client.cert on May 29th (~ 10 days before cert expired). This did not happen on the controller, but does not seem to be using them (see below table)

All seems to be up and going, although I might be in for some more trouble

ziticontroller.thesmithcave.nz:1280 = expires 24
ziticontroller.thesmithcave.nz:6262 = expires 24
ziticontroller.thesmithcave.nz:3022 = expired

even though I can still connect through that gateway and showing as online. Not sure that should be correct but is working at the moment.

So, should certificates be auto-renewing? I dont have logs as lost when restarting docker. Is this likely a bug.

[TheLumberjack]

@gooseleggs

So, should certificates be auto-renewing

As you saw, routers definitely should auto renew. I don’t think the controller was ever setup to do it, so it’s probably more a feature request for the controller than a bug per-se, but it seems like a nice/friendly thing to do. With HA coming, this might already be accounted for, I’ll mention it to the team and see what the consensus is.

[gooseleggs]

Thanks @TheLumberjack. I might log an issue ticket. Not sure if you are still doing monthly get togethers, but definitely something that I think needs to be included. Especially as there is no easy way of getting notified of upcoming outage.

Done: Feature Request: Auto renewal of controller (and any other) certificates · Issue #1155 · openziti/ziti · GitHub

[johnalotoski]

We’re coming up on a year for some ziti clusters and cert rotation and potential downtime is a concern for us also. So this info is a big help, thanks!

With this thread, we have instructions to re-create / rotate required controller certs that were generated in the quickstart with a relatively short term default (1 yr).

The router cert doesn’t look like an issue because as mentioned above it auto-updates as long as controller certs are valid.

For ziti clients (ex: deployed ziti-edge-tunnelers) which had JWT credentials manually generated from ziti-admin-console (ZAC), the resulting json identity files look like the embedded certs also default to a year, and from what I understand, they will require manual renewal/rotation (please correct me if I’m wrong) to avoid downtime at the expiration date.

I’m not using the latest ZAC but instead from a commit a few months ago. I’ll have to upgrade soon. If this isn’t already included in ZAC, here’s an idea: show a column in the “Manage identity” window for identity expiration. Right now there is a “Created” column with the created date, so I suppose I can just add a year to everything to know the expiration if that assumption is valid.

Another [maybe better] idea would be to have a dedicated page in ZAC for all ziti related certs with expiration that an admin can check at a glance to know what type of required actions might be coming up for cert rotation.

That would be easier for most compared to the poking around that may otherwise need to be done to discover which expiration problems are about to happen.

[TheLumberjack]

@johnalotoski

the resulting json identity files look like the embedded certs also default to a year, and from what I understand, they will require manual renewal/rotation

I need to verify this but I am 90% sure that by default, ziti does NOT enforce certificate expiration. I’m also pretty sure if you want to, you can change openziti to respect the certificate expiration but as you mention that can possibly be a problem, so it’s not enabled by default.

I appreciate the ZAC ideas. I’ll make sure the team sees it and we’ll see what we can do around this.

It’ll probably be a few days before I can confirm the above, but I’m pretty sure that’s how it works.

[gooseleggs]

@johnalotoski

For ziti clients (ex: deployed ziti-edge-tunnelers) which had JWT credentials manually generated from ziti-admin-console (ZAC), the resulting json identity files look like the embedded certs also default to a year, and from what I understand, they will require manual renewal/rotation (please correct me if I’m wrong) to avoid downtime at the expiration date.

The certificates for the clients will continue to work, even after the year expiry. That is what I am seeing.

[johnalotoski]

Thanks for the responses! Sounds like we may be good all around then in preparing for cert rotations and likely not needing to rotate the manually set up clients if we don’t want to. If some caveat comes up we still have at least a few weeks before anything starts expiring, so plenty of time still.

Happy July 1st / July 4th, etc, to those having a holiday weekend!

[gooseleggs]

Been thinking about this, and put in a feature request (Feature Request: Banner for certificate expiry · Issue #148 · openziti/ziti-console · GitHub) to help notify people. Currently logged under ZAC, but would want that for the console as well.