Hello,
We are seeing the same symptoms with a three-controller OpenZiti 2.0.1 cluster deployed using the official ziti-controller Helm chart 3.2.1. We use cert-manager 1.19.2 and trust-manager 0.24.0, following the PKI setup documented by the chart.
We use an Entra ID external JWT signer with JIT enrollment enabled for both certificate and token modes. We reproduce the issue when clients use User Session mode: authentication succeeds and the clients show their services as available, but no traffic goes through.
The edge router logs:
invalid client certificate for api session
We followed the HA PKI layout documented by the official v3 Helm chart. Controller 1 creates the shared edge root, while each controller has its own intermediate edge signer issued by that root.
Controllers 2 and 3 explicitly reference controller 1’s root issuer:
edgeSignerPki:
alternativeIssuer:
name: ziti-controller1-edge-root-issuer
kind: Issuer
group: cert-manager.io
To be fully transparent, I did not diagnose this issue or write the workaround myself. Codex identified the likely certificate-chain problem, proposed rebuilding the chains at controller startup, and generated the wrapper below.
I applied it to our environment and validated the results, but I only understand the general principle and not every internal detail of the router’s certificate validation.
So please correct the analysis if anything is wrong.
As I understand the cert-manager tls.crt files used by identity.cert and identity.server_cert contain the controller leaf and its intermediate, but not the shared root. The shared root is available separately in the trust-manager bundle.
Cert-manager omits the root intentionally ( Certificate resource - cert-manager Documentation ), but in this particular OpenZiti path it seems that the router can end up validating against the intermediate associated with one controller instead of the common root.
That would explain why an API Session Certificate issued by one controller works, while a new one issued by another controller after reconnection or failover is rejected.
The generated workaround builds the following chain when each controller starts:
controller leaf → controller intermediate → shared root
It then starts the controller with a temporary configuration referencing the completed chains.
image:
command:
- /bin/bash
- -ceu
args:
- |
runtime_dir=/tmp/ziti-controller-runtime
source_config=/etc/ziti/ziti-controller.yaml
ca_bundle=/etc/ziti/ctrl-plane-cas/ctrl-plane-cas.crt
client_cert=/etc/ziti/ctrl-plane-client-identity/tls.crt
server_cert=/etc/ziti/ctrl-plane-identity/tls.crt
for required_file in "${source_config}" "${ca_bundle}" "${client_cert}" "${server_cert}"; do
if [[ ! -s "${required_file}" ]]; then
echo "required controller PKI file is missing or empty: ${required_file}" >&2
exit 1
fi
done
mkdir -p "${runtime_dir}"
# The trust-manager bundle is ordered from issuer to trust anchor. Keep
# its final certificate, which is the shared self-signed root.
awk '
/-----BEGIN CERTIFICATE-----/ { certificate = ""; in_certificate = 1 }
in_certificate { certificate = certificate $0 ORS }
/-----END CERTIFICATE-----/ { last_certificate = certificate; in_certificate = 0 }
END { printf "%s", last_certificate }
' "${ca_bundle}" > "${runtime_dir}/shared-root.crt"
root_subject="$(openssl x509 -in "${runtime_dir}/shared-root.crt" -noout -subject -nameopt RFC2253)"
root_issuer="$(openssl x509 -in "${runtime_dir}/shared-root.crt" -noout -issuer -nameopt RFC2253)"
if [[ "${root_subject#subject=}" != "${root_issuer#issuer=}" ]]; then
echo "the last certificate in ${ca_bundle} is not a self-signed root" >&2
exit 1
fi
cat "${client_cert}" "${runtime_dir}/shared-root.crt" > "${runtime_dir}/client-chain.crt"
cat "${server_cert}" "${runtime_dir}/shared-root.crt" > "${runtime_dir}/server-chain.crt"
sed \
-e "s|${client_cert}|${runtime_dir}/client-chain.crt|" \
-e "s|${server_cert}|${runtime_dir}/server-chain.crt|" \
"${source_config}" > "${runtime_dir}/ziti-controller.yaml"
exec ziti controller run "${runtime_dir}/ziti-controller.yaml" "$@"
# bash uses the first argument after the script as $0. The controller
# chart appends the cluster agent flags after this placeholder.
- ziti-controller-chain-wrapper
Since applying it, Windows and macOS User Sessions work fine, repeated tunnel reconnections continue to work, and connectivity still works after transferring leadership to another controller.
So, Does this look like the Helm chart manifestation of #4094 ( Edge router rejects valid first-party client certificates when the edge signing CA differs from the ctrl-channel CA · Issue #4094 · openziti/ziti · GitHub ), or is it still related to the shared-root case in #3857 ( Router rejects API session certificate after HA failover for ext-jwt identities without enrollment · Issue #3857 · openziti/ziti · GitHub )?
Is a fix planned, and would it be useful to open a separate issue in openziti/ziti or openziti/helm-charts?
Thanks