Hi Again,
I've been playing with 3rd party CA auto enrolment on my HA system which i understand should automatically create the identity upon client enrolment.
I have been able to create and verify the 3rd party CA no problem.
I was able to enrol a client device using ZET along with the 3rd party client cert, key and re-usable JWT.
But when i checked ziti edge list identities the identity entry was only listed on the Controller that the client enrolled to. The other two HA Controllers do not show the identity and therefore when the client attempts to auth against one of them, auth fails.
I guess integration of this feature might still be outstanding for HA ?
Thanks in advance!
3rd party CAs should work in HA, however, reviewing our existing tests, I don't see where they are being exercised on a regular cadence.
The underlying structures (ca, identities, authenticators) used in 3rd party CA enrollment and authentication should all synchronize through HA controllers. Not seeing an identity in some of your controller but seeing it in others seems to indicate a connection problem between controllers. I suggest reviewing your logs on the controllers w/o the identity to ensure they are connecting to the other controllers successfully.
I will take time to ensure there are no other issues by standing up a 3 controller HA environment, adding a ca, enrolling against that CA via ottca and then authenticating against the 3 controllers individually. Let me know if there are any other details I should take into account for reproduction.
Thanks @andrew.martinez.
My HA system appears to be operating as expected apart from this. I'm able to create other identities using ziti edge create identity ... and see the config synced to the other controllers in the cluster. I'm able to enrol them using ZET and they work as expected.
Here is an all in one bash script for Debian based on the HA quickstart that you can use to reproduce the issue. I just run it on a "Vanilla" Debian Bookworm VM.
It creates a 2 Controller cluster, adds the 3p CA, creates test-client-1 client certificate/key, then enrols using ZET.
At the end it does ziti edge list identities against both HA Controllers to demonstrate that the auto created identity only exists on ctrl1
ZITI_CLI_DEB_VER=1.5.4
ZITI_CONTROLLER_DEB_VER=1.5.4
ZITI_ROUTER_DEB_VER=1.5.4
# Install OZ packages
apt-get update
apt-get install -y gnupg curl wget
curl -sSLf https://get.openziti.io/tun/package-repos.gpg | gpg --dearmor --output /usr/share/keyrings/openziti.gpg
chmod a+r /usr/share/keyrings/openziti.gpg
echo "deb [signed-by=/usr/share/keyrings/openziti.gpg] https://packages.openziti.org/zitipax-openziti-deb-test debian main" > /etc/apt/sources.list.d/openziti-release.list
apt-get update
apt-get install -y openziti=${ZITI_CLI_DEB_VER} openziti-controller=${ZITI_CONTROLLER_DEB_VER} openziti-router=${ZITI_ROUTER_DEB_VER} openziti-console --allow-downgrades
# Set up HA ctrl1
export TRUST_DOMAIN="example.trust.domain"
export ZITI_PWD="password"
export ZITI_INST="ctrl1"
export ZITI_CTRL_PORT="6400"
export ZITI_ROUTER_PORT="6401"
export ZITI_INITIAL_CTRL="tls:ctrl1.${TRUST_DOMAIN}:${ZITI_CTRL_PORT}"
echo "127.0.0.1 ctrl1.${TRUST_DOMAIN}" >> /etc/hosts
echo "127.0.0.1 ctrl2.${TRUST_DOMAIN}" >> /etc/hosts
echo "127.0.0.1 rout1.${TRUST_DOMAIN}" >> /etc/hosts
ziti edge quickstart ha \
--instance-id="ctrl1" \
--ctrl-port="${ZITI_CTRL_PORT}" \
--router-port="${ZITI_ROUTER_PORT}" \
--home="/sharedfs/ziti" \
--ctrl-address="${ZITI_INST}.${TRUST_DOMAIN}" \
--router-address="${ZITI_INST}.${TRUST_DOMAIN}" \
--trust-domain="${TRUST_DOMAIN}" \
--password $ZITI_PWD \
&> ctrl1.log &
echo "Log: ./ctrl1.log"
# Wait for ctrl1 to finish setup
sleep 30
# Set up HA ctrl2
export TRUST_DOMAIN="example.trust.domain"
export ZITI_PWD="password"
export ZITI_INST="ctrl2"
export ZITI_INITIAL_CTRL="tls:ctrl1.${TRUST_DOMAIN}:6400"
export ZITI_CTRL_PORT="6500"
export ZITI_ROUTER_PORT="6501"
mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/keys"
mkdir -p "/tmp/${ZITI_INST}/pki/root-ca/certs"
cp /sharedfs/ziti/pki/root-ca/keys/root-ca.key /tmp/${ZITI_INST}/pki/root-ca/keys/
cp /sharedfs/ziti/pki/root-ca/certs/root-ca.cert /tmp/${ZITI_INST}/pki/root-ca/certs/
cp /sharedfs/ziti/pki/root-ca/index.txt /tmp/${ZITI_INST}/pki/root-ca/index.txt
ziti edge quickstart join \
--instance-id "${ZITI_INST}" \
--ctrl-port "${ZITI_CTRL_PORT}" \
--router-port "${ZITI_ROUTER_PORT}" \
--home "/tmp/${ZITI_INST}" \
--ctrl-address="${ZITI_INST}.${TRUST_DOMAIN}" \
--router-address="${ZITI_INST}.${TRUST_DOMAIN}" \
--trust-domain="${TRUST_DOMAIN}" \
--cluster-member "${ZITI_INITIAL_CTRL}" \
--password $ZITI_PWD \
&> ctrl2.log &
echo "Log: ./ctrl2.log"
# Wait for ctrl2 to finish setup
sleep 30
# 3rd Party CA creation
CA_COMMON_NAME="ziti_3p_CA"
ZITI_3P_PKI="${HOME}"
create_client_cert() {
local client_name=$1
ziti pki create client \
--pki-root="${ZITI_3P_PKI}" \
--ca-name=${CA_COMMON_NAME} \
--client-name=${client_name} \
--client-file=${client_name}
}
ziti edge login -u admin -p ${ZITI_PWD} localhost:6400 --yes
ziti pki create ca --pki-root="${ZITI_3P_PKI}" --ca-file "${CA_COMMON_NAME}" --ca-name "${CA_COMMON_NAME}"
ziti edge create ca ${CA_COMMON_NAME} "$ZITI_3P_PKI/${CA_COMMON_NAME}/certs/${CA_COMMON_NAME}.cert" --auth=true --autoca=true --identity-name-format="[commonName].${CA_COMMON_NAME}.identity"
caid=$(ziti edge list cas 'name = "'${CA_COMMON_NAME}'"' -j | jq -r '.data[0].id')
identity_to_verify=$(ziti edge list cas 'name = "'${CA_COMMON_NAME}'"' -j | jq -r '.data[0].verificationToken')
create_client_cert $identity_to_verify
verification_cert=$ZITI_3P_PKI/$CA_COMMON_NAME/certs/$identity_to_verify.cert
verification_key=$ZITI_3P_PKI/$CA_COMMON_NAME/keys/$identity_to_verify.key
ziti edge verify ca ${CA_COMMON_NAME} --cert $verification_cert
auth_token=$(jq -r .edgeIdentities.default.token $HOME/.config/ziti/ziti-cli.json)
curl -sk -H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "zt-session: $auth_token" \
https://ctrl1.${TRUST_DOMAIN}:6400/edge/management/v1/cas/${caid}/jwt > "$ZITI_3P_PKI/$CA_COMMON_NAME/auto.jwt"
create_client_cert "test-client-1"
# ZET install
UBUNTU_LTS=jammy
echo "deb [signed-by=/usr/share/keyrings/openziti.gpg] https://packages.openziti.org/zitipax-openziti-deb-test $UBUNTU_LTS main" | tee /etc/apt/sources.list.d/openziti.list > /dev/null
rm -f /usr/share/keyrings/openziti.gpg
curl -sSLf https://get.openziti.io/tun/package-repos.gpg | gpg --dearmor --output /usr/share/keyrings/openziti.gpg
chmod -c +r /usr/share/keyrings/openziti.gpg
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y install ziti-edge-tunnel="1.5.12" --allow-downgrades
systemctl enable --now ziti-edge-tunnel.service
# ZET enrolment using 3p CA client cert/key and re-usable JWT.
ziti-edge-tunnel enroll -j $ZITI_3P_PKI/$CA_COMMON_NAME/auto.jwt -c $ZITI_3P_PKI/$CA_COMMON_NAME/certs/test-client-1.cert -k $ZITI_3P_PKI/$CA_COMMON_NAME/keys/test-client-1.key -i /opt/openziti/etc/identities/test-client-1.json
# Login to ctrl1 and list identities
ziti edge login -u admin -p ${ZITI_PWD} localhost:6400 --yes
ziti edge list identities
# Login to ctrl2 and list identities
ziti edge login -u admin -p ${ZITI_PWD} localhost:6500 --yes
ziti edge list identities
# Manually create another identity using ziti CLI
ziti edge create identity "test-client-2.identity" --role-attributes ssh -o "./test-client-2.identity.jwt"
# Login to ctrl1 and list identities
ziti edge login -u admin -p ${ZITI_PWD} localhost:6400 --yes
ziti edge list identities
# Login to ctrl2 and list identities
ziti edge login -u admin -p ${ZITI_PWD} localhost:6500 --yes
ziti edge list identities
I have discovered a set of use cases that are not dispatching events properly in HA which make using 3rd party auto ca enrollment problematic. See this issue for tracking: Auto CA enrollments do not dispatch events in HA · Issue #3068 · openziti/ziti · GitHub
I am working on a fix as of right now.
1 Like