PSA - quickstart change for latest tunnelers and c/python/swift/node sdk

TheLumberjack · August 21, 2023, 4:45pm

Hi all,

UPDATED Aug 22 2023

Just wanted to drop a quick note about a bug in the quickstart. If you have generated your network from the quickstart recently. You want to verify the setting in the web section of your controller to ensure newer tunnelers don’t have connectivity issues.

These commands should work with most/many/all the previous versions of the quicsktart but I can’t really test them all. If you run into issues, comment back on this post.

Update Your Controller Config

source the .env file (or figure out how to set the variables accordingly):
```
source $HOME/.ziti/quickstart/$(hostname -s)/$(hostname -s).env
```

Run this command. It will tell you whether or not you need to update your controller’s config file:

echo ""
if [[ 0 -lt $(grep -c "${ZITI_PKI}/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}/certs/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}.cert" $ZITI_HOME/$ZITI_NETWORK.yaml) ]]
then
  echo "You need to update/edit this file"
else
  echo "Your quickstart is up-to-date/accurate"
fi

If you see: Your quickstart is up-to-date/accurate, you’re done.

If you see: You need to update/edit this file, run the commands listed below to repair your quickstart:

sed -i 's#'"${ZITI_PKI}/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}/certs/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}.cert"'#'"${ZITI_PKI}/${ZITI_PKI_CTRL_EDGE_ROOTCA_NAME}/certs/${ZITI_PKI_CTRL_EDGE_ROOTCA_NAME}.cert"'#g' $ZITI_HOME/$ZITI_NETWORK.yaml

echo "adding controller root CA to ca bundle: $ZITI_PKI/$ZITI_PKI_CTRL_ROOTCA_NAME/certs/$ZITI_PKI_CTRL_ROOTCA_NAME.cert"
cat "$ZITI_PKI/$ZITI_PKI_CTRL_ROOTCA_NAME/certs/$ZITI_PKI_CTRL_ROOTCA_NAME.cert" > "${ZITI_PKI_CTRL_CA}"

echo "adding signing root CA to ca bundle: $ZITI_PKI/$ZITI_PKI_SIGNER_ROOTCA_NAME/certs/$ZITI_PKI_SIGNER_ROOTCA_NAME.cert"
cat "$ZITI_PKI/$ZITI_PKI_SIGNER_ROOTCA_NAME/certs/$ZITI_PKI_SIGNER_ROOTCA_NAME.cert" >>"${ZITI_PKI_CTRL_CA}" >> "${ZITI_PKI_CTRL_CA}"

echo "adding secondary signing intermediate into ca bundle: $ZITI_PKI/$ZITI_PKI_SIGNER_ROOTCA_NAME/certs/${ZITI_PKI_SIGNER_INTERMEDIATE_NAME}_spurious_intermediate.cert"
cat "$ZITI_PKI/$ZITI_PKI_SIGNER_ROOTCA_NAME/certs/${ZITI_PKI_SIGNER_INTERMEDIATE_NAME}_spurious_intermediate.cert" >> "${ZITI_PKI_CTRL_CA}"

Restart the Controller

If you’re using systemd, restart the controller: sudo systemctl restart ziti-controller

Verify the Controller Config

You can now use openssl to verify your controller is correctly updated. Run the following two commands. The first one will pull the cabundle openssl command:

curl -sk https://${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:${ZITI_CTRL_EDGE_ADVERTISED_PORT}/.well-known/est/cacerts \
  | openssl base64 -d \
  | openssl pkcs7 -inform DER -outform PEM -print_certs -out /tmp/well.known.cacerts.pem

openssl s_client \
  -connect ${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:${ZITI_CTRL_EDGE_ADVERTISED_PORT} \
  -CAfile /tmp/well.known.cacerts.pem < /dev/null 2>&1 \
  | grep "Verify return code"

If you see anything other than: Verify return code: 0 (ok) you still have a problem. Revisit these steps.

kashif · August 24, 2023, 9:55am

TheLumberjack:

openssl s_client \
  -connect ${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:${ZITI_CTRL_EDGE_ADVERTISED_PORT} \
  -CAfile /tmp/well.known.cacerts.pem < /dev/null 2>&1 \
  | grep "Verify return code"

Thank you very much.
System updated.

jptechnical · August 25, 2023, 7:25pm

Is this acceptable output?

grep: /.yaml: No such file or directory
Your quickstart is up-to-date/accurate

I forgot, I am running this in docker compose

TheLumberjack · August 25, 2023, 7:26pm

The script assumes you've sourced the .env file so that the variables are available. Make sure you source $HOME/.ziti/$(hostname -s)/$(hostname -s).env. ~~I'll update the instructions. thx~~ Turns out it was step one of the instructions Make sure you source that file.

Oh docker-compose... You'll want to exec into the container to source the file. Then you can source it. I'll try it now to make sure it works.

jptechnical · August 25, 2023, 7:29pm

TheLumberjack:

echo ""
if [[ 0 -lt $(grep -c "${ZITI_PKI}/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}/certs/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}.cert" $ZITI_HOME/$ZITI_NETWORK.yaml) ]]
then
  echo "You need to update/edit this file"
else
  echo "Your quickstart is up-to-date/accurate"
fi

So I tried this two ways.

I sourced the .env file in my docker project folder, ran the command, output as shown.
I jumped into a shell on my controller, sourced the ziti.env file, got a tone of NOT OVERRIDING... messages which tells me it's already set and in the environment. Ran the command, same output.

TheLumberjack · August 25, 2023, 7:33pm

Sorry. Thaks for pointing out docker is different. Here's the working docker compose instructions. I confirmed the compose steps are, unfortunately different. I'll update them with the updated steps.

docker exec -it docker-compose-ziti-controller-1 bash (or the correct name)

run:

if [[ 0 -lt $(grep -c "${ZITI_PKI}/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}/certs/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}.cert" $ZITI_HOME/ziti-controller.yaml) ]]
then
  echo "You need to update/edit this file"
else 
  echo "Your quickstart is up-to-date/accurate"
fi

Looks like the sed command to fix it will change too. Let me get these instructions up to date for docker...

jptechnical · August 25, 2023, 7:37pm

Interestingly, that still didn't work. Here are the particulars about my setup... I notice that my .yaml file is named after the host.

$ grep -c "${ZITI_PKI}/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}/certs/${ZITI_PKI_CTRL_EDGE_INTERMEDIATE_NAME}.cert" ziti.mydomain.com.yaml
0

But the .env isn't correct, so far as I can tell.

$ echo $ZITI_HOME/$ZITI_NETWORK.yaml
/persistent/ziti.yaml

The actual file is ziti.mydomain.com.yaml

$ ll
total 68
drwxrwxrwx 4 root root  4096 Aug 15 14:51 ./
drwxr-xr-x 1 root root  4096 Aug 15 14:50 ../
-rw-r--r-- 1 ziti ziti     0 Aug 15 14:51 access-control.init
drwxr-xr-x 2 ziti ziti  4096 Aug 15 14:51 db/
drwxr-xr-x 9 ziti ziti  4096 Aug 15 14:51 pki/
-rw-r--r-- 1 ziti ziti 10046 Aug 15 14:51 ziti.env
-rw-r--r-- 1 ziti ziti    19 Aug 15 14:51 ziti.mydomain.com-87ab719dd0f7.init
-rw------- 1 ziti ziti 10544 Aug 15 14:51 ziti.mydomain.com.cas
-rw------- 1 ziti ziti  4163 Aug 15 14:51 ziti.mydomain.com.cert
-rw------- 1 ziti ziti  1006 Aug 15 14:51 ziti.mydomain.com.jwt
-rw------- 1 ziti ziti  3243 Aug 15 14:51 ziti.mydomain.com.key
-rw-r--r-- 1 ziti ziti     0 Aug 15 14:51 ziti.mydomain.com.log
-rw------- 1 ziti ziti  2065 Aug 15 14:51 ziti.mydomain.com.server.chain.cert
-rw-r--r-- 1 ziti ziti  1726 Aug 15 14:51 ziti.mydomain.com.yaml

edit: I did an sed on the actual domain.

edit again: I think the issue is that doing this from a docker instance will end up in some inconsistencies in the .env file. The .env file in the container is, I assume, derived from the .env file you use on docker compos up... maybe not. It definitely isn't the same file, because the ziti.env file in the container is full of if statements.

At any rate, being able to grep the file was sufficient for me. Knowing that in the /persistent folder there is a your.controller.name.yaml file and that needs to be grepped for the literal string is sufficient for me. Probably would be if someone chooses to go all cowboy and run it in docker anyway.

TheLumberjack · August 25, 2023, 8:00pm

Yeah, there are definitely differences in each quickstart that need to be accounted for. @qrkourier was working through that and should be putting up an updated set of steps soon. He let me know he was working on it.

qrkourier · August 25, 2023, 8:05pm

Yep, working on a checker/repairer script.

It's unnecessary to source the env file with Docker Compose because user "ziti" ~/.bashrc already sources it when you exec bash to log in. I didn't check if that's also the case for the Docker quickstart (without Compose), but I'm guessing it is the case.

For Docker and Docker Compose quickstart users, any checking/repairing for this cert chain problem must be done as user "ziti" in the controller's or router's container.

jptechnical · August 25, 2023, 9:51pm

Regardless of the script, if the grep command finds the string, then that confirms that no action is required, is that correct? The script certainly is helpful, but knowing what it's checking for as either a footnote or a description of the problem would be just as valuable, IMO. In such a case, if there is a option to manually verify then this would cover the edge cases where the script doesn't work and the end user doesn't need to understand what the script is doing.

qrkourier · August 25, 2023, 10:36pm

That will detect one of the problems, but a grep of the controller config doesn't tell the whole story.

The thing to test is whether or not each TLS server presents a partial server cert chain that can be verified (completed) with cert(s) from CA bundle.

Let's start with how to check for a verification problem. The script will help document and automate how to repair the cert chains if necessary. You can perform these tests from the host or inside the controller container if you're using one of the Docker quickstarts.

First, you need a copy of the CA bundle as a PEM file.

# adjust IP and port to match the controller's client API web listener
  curl -sSk "https://127.0.0.1:1280/.well-known/est/cacerts" \
    | openssl base64 -d \
    | openssl pkcs7 -inform DER -outform PEM -print_certs -out "/tmp/well-known-anchors.pem"

Next, download a copy of the partial chain that's presented by each TLS server to TLS clients. Do this for the client API (e.g., :1280), ctrl plane (e.g., :6262), and public router edge (e.g., :3022)

openssl s_client -connect 127.0.0.1:6262 -showcerts </dev/null > /tmp/server-chain.pem 2>/dev/null

Finally, verify the server cert chain is completed by the CA bundle.

openssl verify -CAfile /tmp/well-known.pem /tmp/server-chain.pem

If you encounter a certificate verification problem with this method:

Fix the verification chain (the script I'm working will help to document and, optionally, automate this)
Re-start the controller or router that failed verification then re-test
As needed, re-enroll the identities and routers that have the old CA bundle

A script for automatically checking TLS server certs are verifiable

This script requires the quickstart's env vars to be set in advance and automates checking those three TLS servers are currently verifiable.

#!/usr/bin/env bash

set -euo pipefail

check_chains(){

  declare -A CHAINS
  : "${CERTSDIR:=$(mktemp --directory --tmpdir="${TMPDIR:-/tmp}" "certs.${DATESTAMP}.XXXX")}"
  CHAINS[ctrl_plane]="${CERTSDIR}/controller-ctrl-chain.pem"
  CHAINS[edge_web]="${CERTSDIR}/controller-edge-web-chain.pem"
  CHAINS[router]="${CERTSDIR}/router-chain.pem"

  # download the controller's trust anchor bundle as currently presented to enrollees
  curl -sSk "https://${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:${ZITI_CTRL_EDGE_ADVERTISED_PORT}/.well-known/est/cacerts" \
    | openssl base64 -d \
    | openssl pkcs7 -inform DER -outform PEM -print_certs -out "${CERTSDIR}/well-known-anchors.pem"

  # download the controller's edge web listener cert chain
  openssl s_client \
    -connect "${ZITI_CTRL_EDGE_ADVERTISED_ADDRESS}:${ZITI_CTRL_EDGE_ADVERTISED_PORT}" \
    -showcerts </dev/null > "${CHAINS[edge_web]}" 2>/dev/null
  # download the controller's ctrl plane listener cert chain
  openssl s_client \
    -connect "${ZITI_CTRL_ADVERTISED_ADDRESS}:${ZITI_CTRL_ADVERTISED_PORT}" \
    -showcerts </dev/null > "${CHAINS[ctrl_plane]}" 2>/dev/null || true  # requires client cert to avoid errexit
  # download the router's edge listener cert chain
  openssl s_client \
    -connect "${ZITI_ROUTER_NAME}:${ZITI_ROUTER_PORT}" \
    -showcerts </dev/null > "${CHAINS[router]}" 2>/dev/null || true  # requires client cert to avoid errexit
  
  for CHAIN in "${!CHAINS[@]}"; do
    if openssl verify -CAfile ${CERTSDIR}/well-known-anchors.pem "${CHAINS[${CHAIN}]}" &>/dev/null; then
      echo "INFO: $CHAIN certificate chain verified."
    else
      echo "ERROR: cert verification failed for $CHAIN with command 'openssl verify -CAfile ${CERTSDIR}/well-known-anchors.pem"\
      "\"${CHAINS[${CHAIN}]}\"'." >&2
      exit 1
    fi
  done

}

DATESTAMP=$(date +%Y%m%d%H%M%S)
check_chains

One-liner to run the above script:

curl -sSf https://raw.githubusercontent.com/openziti/ziti/add-signer-grandparent-ziti-env/quickstart/check-repair-cert-chains.bash \
    | bash

I pulled this check_chains() function from the script I'm working on for automating repairs. That's got more edges. If you can start over with a fresh quickstart v0.30.1 or newer, it won't have this problem.

The next challenge with the repair script I'm working on is what to do with quickstart routers that were enrolled with incomplete chains. Recreating them seems necessary, but a step-by-step procedure for that hasn't emerged yet. I tried ziti router run --extend but that didn't seem to work. After solving routers I plan to look at whether existing identities need to be re-created.

jptechnical · August 26, 2023, 12:19am

Edit: @qrkourier is the docker quickstart:latest (which shows 0.30.1) the current version which resolves this issue? Just confirming.

I had to tweak the output, but it looks like my install is affected. I think I will blow it away and start fresh. I plan on making this one a pet... so I would like to start with a clean install.

 for CHAIN in "${!CHAINS[@]}"; do
    if openssl verify -CAfile ${CERTSDIR}/well-known-anchors.pem "${CHAINS[${CHAIN}]}" &>/dev/null; then
      echo "INFO: $CHAIN certificate chain verified."
    else
      echo "ERROR: cert verification failed for $CHAIN with command openssl verify -CAfile ${CERTSDIR}/well-known-anchors.pem" ${CHAINS[${CHAIN}]}\"
      exit 1
    fi
  done

My output is

# ./checkcert.sh
ERROR: cert verification failed for router with command openssl verify -CAfile /tmp/certs.20230826001342.jTJ1/well-known-anchors.pem /tmp/certs.20230826001342.jTJ1/router-chain.pem"

btw, I was getting this error...

# ./checkcert.sh
./checkcert.sh: line 36: unexpected EOF while looking for matching `"'

qrkourier · August 26, 2023, 1:38pm

This issue was first corrected in 0.30.0, and 0.30.1 is currently latest.

curl -sSf https://registry.hub.docker.com/v2/repositories/openziti/quickstart/tags/ \
    | jq '.results[]|select(.name|match("^(latest|0.30.1)$"))|[.name, .digest]'

[
  "latest",
  "sha256:2f14bb593d33f924422fcf25e592e7fffad579681a34e8ef761ba87c2cd2a46f"
]
[
  "0.30.1",
  "sha256:2f14bb593d33f924422fcf25e592e7fffad579681a34e8ef761ba87c2cd2a46f"
]

Thanks for testing the checker script! I'll add that missing double quote to the original post.

Here's a one-liner for running the checker script that assumes the quickstart env vars are defined by either logging in to the quickstart container (exec bash) or sourcing the ziti.env file.

curl -sSf https://raw.githubusercontent.com/openziti/ziti/add-signer-grandparent-ziti-env/quickstart/check-repair-cert-chains.bash \
    | bash

Topic		Replies	Views
I think quickstart:0.30.1 broke something in the edge-tunnel connections	18	406	September 1, 2023
Is quickstart working right now? (TAG 0.30.3-0ca6e6d7)	6	186	September 28, 2023
Reinstalling the Quickstart program Building/Development	7	233	March 31, 2022
Let's Encrypt on Controller General Questions	3	557	February 9, 2023
PSA: Major Quickstart Changes in v0.28.2 Support	0	140	June 22, 2023

PSA - quickstart change for latest tunnelers and c/python/swift/node sdk

Update Your Controller Config

Restart the Controller

Verify the Controller Config

A script for automatically checking TLS server certs are verifiable

Related topics