Creating certs for a remote private router

Ok. I got to the bottom of the issue. It comes down to a few things but mainly it’s because the command I provided to @markamind on his other post was ever so slightly incorrect and that caused a lot of confusion! :frowning: I’ll update that post after this one.

@arslane gave me access to his OCI instance saving me a bit of time making an OCI machine. I wanted to use it anyway, to save you the time and so you’ll have your own instance that’s configured correctly as reference. Here’s what I needed to do.

Check all the ports

There are four main ports needed to add a second router to a quickstart-enabled instance. The first two are always opened as part of the quickstart, the second two are only opened when you add a second router. These are those ports:

  • 8441 - the client/management API port used in the quickstarts
  • 8442 - the edge router port that accepts edge connections used in the quickstarts
  • 6262 - the controller’s control plane port - routers connect to the controller on this port
  • 10080 - the port the quickstart uses for the edge router’s link listener

First thing I did was to check that all these “server” ports all returned a certificate with DNS names properly setup. Port 6262 was still returning the IP in the DNS field. That was when I realized that the instructions I gave to @markamind were incomplete.

Create TWO new certs

The quickstart comes with three PKI’s setup for the controller/edge controller as it’s meant to be educational. There’s a PKI for the overall network - the ‘controller’ (port 6262). One for the edge controller which is not quite the same as the ‘controller’. This is the PKI for the APIs (port 8441). Both certs need to be fixed. To do that I sourced the .env file that comes with the quickstart:

. $HOME/.ziti/quickstart/$(hostname)/$(hostname).env

Then I setup the variables but notice I did the DNS-related ones differently. Also note that I have two different files called out at the bottom of this code block… one for each cert we’re about to make. It’s vital that you do not set the DNS related fields to an IP address. Here you can see I made up one, and used the hostname for the other:

now="$(date +'%Y-%m-%d_%H%M%S')"

EDGE_CONTROLLER_EXTERNAL_DNS_NAME=my.private.ziti.controller
EDGE_CONTROLLER_PRIVATE_DNS_NAME=$(hostname)
EDGE_CONTROLLER_EXTERNAL_IP_ADDRESS=144.24.195.21
EDGE_CONTROLLER_PRIVATE_IP_ADDRESS=10.0.0.235

pki_allow_list_dns="${EDGE_CONTROLLER_EXTERNAL_DNS_NAME},${EDGE_CONTROLLER_PRIVATE_DNS_NAME},localhost,$(hostname)"
pki_allow_list_ip="127.0.0.1,${EDGE_CONTROLLER_EXTERNAL_IP_ADDRESS},${EDGE_CONTROLLER_PRIVATE_IP_ADDRESS}"

new_ctrl_cert_name="${ZITI_CONTROLLER_HOSTNAME}-${now}"
new_edge_ctrl_cert_name="${ZITI_EDGE_CONTROLLER_HOSTNAME}-${now}"

generated a new server cert for the controller by running:

"${ZITI_BIN_DIR}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name ${ZITI_CONTROLLER_INTERMEDIATE_NAME} \
  --key-file "${ZITI_CONTROLLER_HOSTNAME}-server" \
  --server-file "${new_ctrl_cert_name}-server" \
  --dns "${pki_allow_list_dns}" --ip "${pki_allow_list_ip}" \
  --server-name "${new_ctrl_cert_name} server certificate"

generated a new server cert for the edge controller by running:

"${ZITI_BIN_DIR}/ziti" pki create server \
  --pki-root="${ZITI_PKI_OS_SPECIFIC}" \
  --ca-name ${ZITI_EDGE_CONTROLLER_INTERMEDIATE_NAME} \
  --key-file "${ZITI_EDGE_CONTROLLER_HOSTNAME}-server" \
  --server-file "${new_edge_ctrl_cert_name}-server" \
  --dns "${pki_allow_list_dns}" --ip "${pki_allow_list_ip}" \
  --server-name "${new_edge_ctrl_cert_name} server certificate"

Finally, we can find and output these two files:

cat <<HERE

    CONTROLLER server cert use:
        $(find $ZITI_HOME -name "*${new_ctrl_cert_name}*chain.pem")
    EDGE CONTROLLER server cert use:
        $(find $ZITI_HOME -name "*${new_edge_ctrl_cert_name}*chain.pem")

HERE

which looked like this:

    CONTROLLER server cert use:
        /home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/instance-20220807-1808-intermediate/certs/instance-20220807-1808-2022-08-07_190024-server.chain.pem
    EDGE CONTROLLER server cert use:
        /home/ubuntu/.ziti/quickstart/instance-20220807-1808/pki/144.24.195.21-intermediate/certs/144.24.195.21-2022-08-07_190024-server.chain.pem

Edit the config file

With those files in hand, now I could edit the controller’s config file:

vi $ZITI_HOME/$(hostname).yaml

The first thing I did was edit the identity.server_cert near the top of the file and replaced the server_cert with the NEW server cert. Then I found the edge controller server_cert location in the web.identity.server_cert section (near the bottom of the file) and I updated that value.

Finally I restarted the controller:

sudo systemctl restart ziti-controller

Making sure it works

I wanted to make sure @arslane would be in good shape, so I make a “clint-aws” edge router:

ziti edge create edge-router clint-aws -o clint-aws.jwt -t

I scp’ed the jwt from @arslane’s instance, and scp’ed it to my AWS router. Then I cd’ed to that $ZITI_HOME, copied my existing config file, changed the necessary settings, enrolled it, and tried to start that router. When I did that the first time, my edge router in AWS reported “link dialing failed”:

certificate is valid for 127.0.0.1, not 144.24.195.21] linkId=[4xwLnaMbpcofMRpOsYnaR6]} link dialing failed

There seems to be another bug with the quickstart though because the edge-router’s config had an IP address in the SANS section. It looked like this:

    sans:
      dns:
        - 144.24.195.21
        - localhost
      ip:
        - "127.0.0.1"

I updated that section to use the hostname in the dns section and to use the external ip in the ip section. It now looks like this:

    sans:
      dns:
        - "instance-20220807-1808"
        - "localhost"
      ip:
        - "127.0.0.1"
        - "144.24.195.21"

The problem is that the certificates for routers are created during the enrollment process. That meant I had to delete the existing edge-router, and recreate it. Using the ziti-cli-functions.sh as a guide, I did that using the following commands:

ZITI_EDGE_ROUTER_RAWNAME="${ZITI_NETWORK}-edge-router"
"${ZITI_BIN_DIR-}/ziti" edge delete edge-router "${ZITI_EDGE_ROUTER_RAWNAME}"
"${ZITI_BIN_DIR-}/ziti" edge create edge-router "${ZITI_EDGE_ROUTER_RAWNAME}" -o "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.jwt" -t -a "public"
"${ZITI_BIN_DIR-}/ziti-router" enroll "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.yaml" --jwt "${ZITI_HOME_OS_SPECIFIC}/${ZITI_EDGE_ROUTER_RAWNAME}.jwt"
sudo systemctl start ziti-router

I verified the ‘public’ edge router started (Arslane’s edge router, it did start) and then went back to my AWS router and started that router. This time it connected to Arslane’s public edge-router. I could verify that by looking at the links formed:

ziti fabric list links
╭────────────────────────┬───────────┬────────────────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID                     │ DIALER    │ ACCEPTOR                           │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │
├────────────────────────┼───────────┼────────────────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 7hx4qaAE1UlL4iPIH51Phg │ clint-aws │ instance-20220807-1808-edge-router │           1 │   65000.0ms │   65000.0ms │ Connected │     up │    130001 │
╰────────────────────────┴───────────┴────────────────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
results: 1-1 of 1

That was a lot more effort than I thought, and would be VERY hard for someone not as familiar with X509/the PKI setup of OpenZiti to figure out!!!

Thanks @arslane for letting me use your OCI instance. I look forward to your follow-up. :slight_smile:

2 Likes