Router connection to Controller, handshake failed

3

Thanks @plorenz.

I have just tried the suggested steps on v1.5.4. It seems i do encounter this issue on reconnects and restarts.

I've also simplified my install process for the purpose of reproducing my issue.

Here's my install script in full, this time it's based on the HA quick start mostly grabbed from here. I just run it on a single Debian VM.

ZITI_CLI_DEB_VER=1.5.4
ZITI_CONTROLLER_DEB_VER=1.5.4
ZITI_ROUTER_DEB_VER=1.5.4

# Install OZ packages
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" --allow-downgrades --allow-remove-essential --allow-change-held-packages -fuy dist-upgrade
apt-get install -y gnupg curl wget
curl -sSLf https://get.openziti.io/tun/package-repos.gpg | gpg --dearmor --output /usr/share/keyrings/openziti.gpg
chmod a+r /usr/share/keyrings/openziti.gpg
echo "deb [signed-by=/usr/share/keyrings/openziti.gpg] https://packages.openziti.org/zitipax-openziti-deb-test debian main" > /etc/apt/sources.list.d/openziti-release.list
apt-get update
apt-get install -y openziti=${ZITI_CLI_DEB_VER} openziti-controller=${ZITI_CONTROLLER_DEB_VER} openziti-router=${ZITI_ROUTER_DEB_VER} openziti-console --allow-downgrades

# Set up HA ctrl1
export TRUST_DOMAIN="example.trust.domain"
export ZITI_PWD="password"
export ZITI_INST="ctrl1"
export ZITI_CTRL_PORT="6400"
export ZITI_ROUTER_PORT="6401"
export ZITI_INITIAL_CTRL="tls:ctrl1.${TRUST_DOMAIN}:${ZITI_CTRL_PORT}"
sudo chown ziti:ziti /sharedfs/
echo "127.0.0.1 ctrl1.${TRUST_DOMAIN}" >> /etc/hosts
echo "127.0.0.1 ctrl2.${TRUST_DOMAIN}" >> /etc/hosts
echo "127.0.0.1 rout1.${TRUST_DOMAIN}" >> /etc/hosts
ziti edge quickstart ha \
    --instance-id="ctrl1" \
    --ctrl-port="${ZITI_CTRL_PORT}" \
    --router-port="${ZITI_ROUTER_PORT}" \
    --home="/sharedfs/ziti" \
    --ctrl-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --router-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --trust-domain="${TRUST_DOMAIN}" \
    --password $ZITI_PWD \
    &> ctrl1.log &

# Wait for ctrl1 to finish setup
sleep 30

# Set up HA ctrl2
export TRUST_DOMAIN="example.trust.domain"
export ZITI_PWD="password"
export ZITI_INST="ctrl2"
export ZITI_INITIAL_CTRL="tls:ctrl1.${TRUST_DOMAIN}:6400"
export ZITI_CTRL_PORT="6500"
export ZITI_ROUTER_PORT="6501"

mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/keys"
mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/certs"
cp /sharedfs/ziti/pki/root-ca/keys/root-ca.key /tmp/${ZITI_INST}/pki/root-ca/keys/
cp /sharedfs/ziti/pki/root-ca/certs/root-ca.cert /tmp/${ZITI_INST}/pki/root-ca/certs/
cp /sharedfs/ziti/pki/root-ca/index.txt /tmp/${ZITI_INST}/pki/root-ca/index.txt

ziti edge quickstart join \
    --instance-id "${ZITI_INST}" \
    --ctrl-port "${ZITI_CTRL_PORT}" \
    --router-port "${ZITI_ROUTER_PORT}" \
    --home "/tmp/${ZITI_INST}" \
    --ctrl-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --router-address="${ZITI_INST}.${TRUST_DOMAIN}" \
    --trust-domain="${TRUST_DOMAIN}" \
    --cluster-member "${ZITI_INITIAL_CTRL}" \
    --password $ZITI_PWD \
    &> ctrl2.log &

# Wait for ctrl2 to finish setup
sleep 30

# Set up router
ziti edge login -p ${ZITI_PWD}

ziti edge create edge-router "edge-router-test-1" --jwt-output-file "/tmp/edge-router-test-1.jwt" --tunneler-enabled

echo "ZITI_CTRL_ADVERTISED_ADDRESS='ctrl1.${TRUST_DOMAIN}'" > /opt/openziti/etc/router/bootstrap.env
echo "ZITI_CTRL_ADVERTISED_PORT='6400'" >> /opt/openziti/etc/router/bootstrap.env
echo "ZITI_ROUTER_ADVERTISED_ADDRESS='rout1.${TRUST_DOMAIN}'" >> /opt/openziti/etc/router/bootstrap.env
echo "ZITI_ROUTER_PORT='3999'" >> /opt/openziti/etc/router/bootstrap.env
echo "ZITI_ENROLL_TOKEN='/tmp/edge-router-test-1.jwt'" >> /opt/openziti/etc/router/bootstrap.env

/opt/openziti/etc/router/bootstrap.bash

systemctl enable --now ziti-router.service


# verify traffic 
ziti ops verify traffic -p ${ZITI_PWD}

After 5 minutes of ziti-router.service uptime i disable the router.

ziti fabric update router edge-router-test-1 --disabled

Confirmed disconnection, Waited 60s, re-enable the router

ziti fabric update router edge-router-test-1 --disabled=false

At this point the router certs have not been renewed.

Now is when i start to see the router connection errors in the controller logs.

{"_context":"tls:0.0.0.0:6500","file":"github.com/openziti/channel/v3@v3.0.39/classic_listener.go:219","func":"github.com/openziti/channel/v3.(*classicListener).acceptConnection.func1","level":"error","msg":"connection handler error for [tls:127.0.0.1:57384] (x509: certificate signed by unknown authority)","time":"2025-04-14T12:03:34.040Z"}