Zrok Controller certificate error when using public CAs on openziti controller

Hello! I have(had) a working zrok self-hosted setup but needed to switch over to using a public CA (Let’s Encrypt) on my openziti controller. Now I am getting this trying to start the controller:

INFO zrok/controller.inspectZiti: inspecting ziti controller configuration
panic: error getting ziti edge client: Post "https://<redacted>:1280/edge/management/v1/authenticate?method=password": x509: certificate signed by unknown authority

The certificate is sound and ziti-router doesn’t have any connection issues.

Also blew everything away for zrok and cleaned out the openziti configurations for ctrl and frontend. Tried to re-run the bootstrap and same error.

I’m going through this process myself in order to answer this question better… Hoping to get through that today. I’ll reply shortly if nobody else hops in

Another piece of the puzzle, attempting to enroll a new identity now

{"level":"info","msg":"http: TLS handshake error from <ip.ip.ip.ip>:64324: remote error: tls: unknown certificate authority","time":"2023-02-10T13:56:31.175Z"}

Feels like another public CA (Let’s Encrypt certs) issue. Some things playing nice, others not so much.

This specific error reminds me of a past problem that I have experienced.

Have you added the public certificate authority to the ziti controller?

There is a video that @TheLumberjack did a while ago.. and includes all of the instructions.. I would be interested to know if this resolves your problem.

I went through all the steps from start to finish enabling zrok in a letsencrypt-enabled openziti. As usual here's a walkthrough video along with all the commands I ran


YouTube Video:


Commands

Prerequisites

  • follow the full quickstart from the openziti docs

  • purchase/provision wildcard DNS entry and verify it resolves

  • obtain a keypair from letsencrypt. example:

    sudo certbot certonly -d '*.clint.demo.openziti.org' --manual --preferred-challenges dns

  • follow the openziti alt server guide

    • controller config, address fields
    • controller config, web.identity section key/server_cert
    • router config, and add alt_server_certs
    • sudo systemctl restart ziti-controller
    • sudo systemctl restart ziti-router

Run as root

currently zrok will end up wanting to find $HOME/.zrok
if you don't want to run as root, keep this in mind

sudo -i


establish a bunch of variables.

I put this into root's .bashrc

export ZROK_ROOT=$HOME/.zrok
export PATH=$PATH:$ZROK_ROOT/bin
export ZROK_CTRL_PORT=18444
export ZROK_FRONTEND_PORT=8080
export ZROK_NGINX_PORT=8445
export ZROK_ZITI_CTRL_WILDCARD=clint.demo.openziti.org
export ZROK_API_ADDRESS="api.${ZROK_ZITI_CTRL_WILDCARD}"
export ZROK_API_ENDPOINT=http://localhost:${ZROK_CTRL_PORT}
export ZROK_ADMIN_TOKEN=my_zrok_admin_token

Nginx

install nginx

sudo apt install nginx -y

backup config file if you want

sudo mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.orig

create a new "zrok-only" config file for nginx

sudo tee /etc/nginx/nginx.conf > /dev/null << HERE
events {
}
http {
  server {
      listen              ${ZROK_NGINX_PORT} ssl;
      server_name         ${ZROK_API_ADDRESS};
      ssl_certificate     /etc/letsencrypt/live/${ZROK_ZITI_CTRL_WILDCARD}/fullchain.pem;
      ssl_certificate_key /etc/letsencrypt/live/${ZROK_ZITI_CTRL_WILDCARD}/privkey.pem;
      ssl_protocols       TLSv1.2;
      ssl_ciphers         HIGH:!aNULL:!MD5;

      location / {
        proxy_pass      http://127.0.0.1:${ZROK_CTRL_PORT};
        error_log       /var/log/nginx/zrok-controller.log;
      }
  }

  server {
      listen              ${ZROK_NGINX_PORT} ssl;
      server_name         *.${ZROK_ZITI_CTRL_WILDCARD};
      ssl_certificate     /etc/letsencrypt/live/${ZROK_ZITI_CTRL_WILDCARD}/fullchain.pem;
      ssl_certificate_key /etc/letsencrypt/live/${ZROK_ZITI_CTRL_WILDCARD}/privkey.pem;
      ssl_protocols       TLSv1.2;
      ssl_ciphers         HIGH:!aNULL:!MD5;

      location / {
        proxy_pass       http://127.0.0.1:${ZROK_FRONTEND_PORT};
        proxy_set_header Host \$host;
        error_log        /var/log/nginx/zrok-frontend.log;
        proxy_busy_buffers_size   512k;
        proxy_buffers    4 512k;
        proxy_buffer_size   256k;
      }
  }
}
HERE

sudo nginx -t
sudo systemctl restart nginx

LetsEncrypt work

add the letsencrypt certificate chain to the ca bundle

cat /etc/letsencrypt/live/${ZROK_ZITI_CTRL_WILDCARD}/fullchain.pem >> $HOME/.ziti/quickstart/ip-172-31-11-231/pki/cas.pem
sudo systemctl restart ziti-controller

echo "verify the controller works properly using the browser and navigating to"
echo ""
echo "  https://ctrl.${ZROK_ZITI_CTRL_WILDCARD}:${ZITI_EDGE_CONTROLLER_PORT}"
echo ""

Logging into ziti

use this command to login to ziti:

export ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT=ctrl.${ZROK_ZITI_CTRL_WILDCARD}:${ZITI_EDGE_CONTROLLER_PORT}
ziti edge login ${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT} -u $ZITI_USER -p $ZITI_PWD -y

Prep for zrok

mkdir -p $ZROK_ROOT/bin
cd $ZROK_ROOT/bin

wget https://github.com/openziti/zrok/releases/download/v0.3.1/zrok_0.3.1_linux_amd64.tar.gz
tar zxvf zrok_0.3.1_linux_amd64.tar.gz

output the ctrl.yml using the variables:

cat > $ZROK_ROOT/ctrl.yml << HERE
v:                  2
admin:
  secrets:
    -               $ZROK_ADMIN_TOKEN
endpoint:
  host:             0.0.0.0
  port:             $ZROK_CTRL_PORT
store:
  path:             zrok.db
  type:             sqlite3
ziti:
  api_endpoint:     "https://${ZITI_EDGE_CTRL_ADVERTISED_HOST_PORT}"
  username:         "${ZITI_USER}"
  password:         "${ZITI_PWD}"
HERE

boostrap zrok

zrok admin bootstrap $ZROK_ROOT/ctrl.yml


ZROK_FRONTEND_TOKEN

look at the logs, find and then set the token

ZROK_FRONTEND_TOKEN="j.mUrwBGrp"

zrok controller service

sudo tee /etc/systemd/system/zrok-controller.service > /dev/null << HERE
[Unit]
Description=zrok-controller
After=network.target

[Service]
User=root
WorkingDirectory=$ZROK_ROOT
ExecStart="$ZROK_ROOT/bin/zrok" controller "$ZROK_ROOT/ctrl.yml"
Restart=always
RestartSec=2
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
HERE

sudo systemctl daemon-reload
sudo systemctl enable --now zrok-controller

create the frontend

zrok admin create frontend ${ZROK_FRONTEND_TOKEN} public https://{token}.${ZROK_ZITI_CTRL_WILDCARD}:${ZROK_NGINX_PORT}

frontend config file

cat > $ZROK_ROOT/http-frontend.yml << HERE
host_match: ${ZROK_ZITI_CTRL_WILDCARD}
address: 0.0.0.0:${ZROK_FRONTEND_PORT}
HERE

sudo tee /etc/systemd/system/zrok-frontend.service > /dev/null << HERE
[Unit]
Description=zrok-frontend
After=network.target

[Service]
User=root
WorkingDirectory=$ZROK_ROOT
ExecStart="$ZROK_ROOT/bin/zrok" access public "$ZROK_ROOT/http-frontend.yml"
Restart=always
RestartSec=2
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target
HERE

sudo systemctl daemon-reload
sudo systemctl enable --now zrok-frontend

Do an invite

zrok invite
(enter your email and submit)

use journalctl to get the registration token:

journalctl -fu zrok-controller

use echo to output some instructions:

export ZROK_REGISTRATION_TOKEN="UrxepUAxiEyk"
echo "To finish the invitation, direct your browser to:"
echo ""
echo "    https://${ZROK_API_ADDRESS}:${ZROK_NGINX_PORT}/register/${ZROK_REGISTRATION_TOKEN}"
echo ""

'enable' zrok

get your enable token from the UI and use it to enable a shell somewhere

ZROK_ENABLE_TOKEN="UIFZ3NeMvFiU"
echo "in a target environment:"
echo "  - disable zrok if needed:"
echo "    zrok disable"
echo ""
echo "  - set the zrok apiEndpoint using the following:"
echo "    zrok config set apiEndpoint https://${ZROK_API_ADDRESS}:${ZROK_NGINX_PORT}"
echo ""
echo "  - enable the zrok env:"
echo "    zrok enable ${ZROK_ENABLE_TOKEN}"

I hope that shows you all the steps, and is clear/easy to follow

1 Like

I forgot to mention in the long list of steps that you need to add the letsencrypt certs to the ca bundle specified in the controller identity.ca. It's this one step in the list:

cat /etc/letsencrypt/live/${ZROK_ZITI_CTRL_WILDCARD}/fullchain.pem >> $HOME/.ziti/quickstart/ip-172-31-11-231/pki/cas.pem
sudo systemctl restart ziti-controller

There were other things along the way which caught me a little bit off guard, which was why I decided to just write down all the steps I took.

That is the one that got me, adding the certs to the bundle!

Thanks @TheLumberjack, you are my hero

1 Like