# HA not working as expected with ext-jwt signer (without JIT enrollment)

**URL:** <https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812>\
**Category:** Uncategorized\
**Created:** [May 6, 2026, 10:57am UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812 "2026-05-06T10:57:06Z")\
**Posts on this page:** 13\
**Page:** 2

<div class="post-metadata">

**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\
**Post date:** [May 26, 2026, 2:24pm UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/22 "2026-05-26T14:24:16Z")

</div>

That isn't an issue, that is Claude not understanding the whole enrollment process and how the PKI is managed. AI LLMs aren't good at system wide scope and context if you don't feed them properly via a prompt or pre-configured architecture or a persona with specific implementation knowledge.

Routers and SDKs that enroll receive a full chain up to but not including the root. The client supplied chain is used as potential links to a trust anchor. In that scenario, the root trust anchor chain will be returned even if the signing controller's intermediate isn't a trust anchor. So what it is saying isn't the issue.

We have a long standing issue to remove that behavior as well, the intermediate signer in the CA pool, but legacy systems and older enrolled clients will start to fail due to a old bug present in 1.3 (I think) that would carry forward for non-HA systems. We added facilities to OpenZiti for systems operators to identify which identities need to either re-enroll or upgrade to an SDK that will auto-refresh its certificates to avoid that situation. Going to HA requires identities to not be in that bugged state.

But saying that, are you using a really old identity? And if so can you send me the identity file w/o private key.

---

<div class="post-metadata">

**Author:** ![marratj](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/marratj/32/3149_2.png) [@marratj](https://openziti.discourse.group/u/marratj)\
**Post date:** [May 31, 2026, 6:52am UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/23 "2026-05-31T06:52:01Z")

</div>

> [@andrew.martinez](#):
>
> But saying that, are you using a really old identity? And if so can you send me the identity file w/o private key.

No, the test cluster I’m using here is a fresh installation with 2.0.0 and the identities I’m using to test are all newly created ones that have been created only after creating the cluster.

---

<div class="post-metadata">

**Author:** ![marratj](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/marratj/32/3149_2.png) [@marratj](https://openziti.discourse.group/u/marratj)\
**Post date:** [June 1, 2026, 7:54am UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/24 "2026-06-01T07:54:05Z")

</div>

So I tried the same again and again, also with the latest stable `2.0.0` release and always see the same behavior. With OTT-enrolled identities, HA and all the failover scenarios work just fine.

But with ext-jwt identities from Keycloak, it falls apart as soon as the initial cluster node is not reachable anymore by the router. Also, adding the identity for any other controller URL than the first one (while all three are up and running) also leads to the router complaining about the invalid client certificate.

The only way I got it to work is my aforementioned "fix" (or workaround) by Claude so that the router adds every root certificate in `self.env.GetRouterId().CaPool().Roots()` to the `rootPool` in `getControllerRootPool` instead of using `self.env.GetRouterId().CA()` and `x509.Verify()` to build the rootPool.

So as it currently stands, we cannot use HA with the stable 2.0.0 release in our environment without applying this crude patch to the router code.

---

<div class="post-metadata">

**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\
**Post date:** [June 4, 2026, 2:50pm UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/25 "2026-06-04T14:50:03Z")

</div>

I will try recreating again in a lab environment and verify each piece's versions again either today or tomorrow.

---

<div class="post-metadata">

**Author:** ![marratj](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/marratj/32/3149_2.png) [@marratj](https://openziti.discourse.group/u/marratj)\
**Post date:** [July 8, 2026, 3:04pm UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/26 "2026-07-08T15:04:07Z")

</div>

Hi Andrew, I saw that you could reproduce the issue in the meanwhile ( [Edge router rejects valid first-party client certificates when the edge signing CA differs from the ctrl-channel CA · Issue #4094 · openziti/ziti · GitHub](https://github.com/openziti/ziti/issues/4094) ). This seems to be the exact issue that we're seeing, as we use the Helm chart to install OpenZiti and that basically uses this divergent identity config you mention there.

The issue with the Helm chart is that it uses cert-manager to build the certificate files instead of the `ziti pki` command and this only builds the file that is then referenced in `server.identity` to be chained to the individual controller's Edge Signer intermediate cert instead of the Root CA from which the Edge Signer intermediate was issued. This is just the way how cert-manager creates the certificate file.

Whereas when you stand up the PKI with `ziti pki`, it creates `server.chain.pem` which includes also the Root CA in the chain to be used for `server.identity`.

This would explain why you couldn't reproduce this issue on your side.

---

<div class="post-metadata">

**Author:** ![mguthrie88](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/mguthrie88/32/151_2.png) [@mguthrie88](https://openziti.discourse.group/u/mguthrie88)\
**Post date:** [July 20, 2026, 2:53pm UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/27 "2026-07-20T14:53:57Z")

</div>

Can you post your helm values in the `edgeSignerPki` block for one of your subsequent controllers? Subsequent controllers should point to the primary controller's root issuer.

```yaml
edgeSignerPki:
  alternativeIssuer:
    # Reference the first node's edge-root-issuer
    name: ziti-controller1-edge-root-issuer
    kind: Issuer
    group: cert-manager.io

```

---

<div class="post-metadata">

**Author:** ![marratj](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/marratj/32/3149_2.png) [@marratj](https://openziti.discourse.group/u/marratj)\
**Post date:** [July 21, 2026, 6:37am UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/28 "2026-07-21T06:37:34Z")

</div>

Yes, they do. Just like it's documented in the Helm chart, the alternativeIssuer on the joining nodes is configured with the root CA of the initial controller. And as you can see above, the bug has been confirmed in the meanwhile: [Edge router rejects valid first-party client certificates when the edge signing CA differs from the ctrl-channel CA · Issue #4094 · openziti/ziti · GitHub](https://github.com/openziti/ziti/issues/4094)

---

<div class="post-metadata">

**Author:** ![ydelafollye](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/ydelafollye/32/4323_2.png) [@ydelafollye](https://openziti.discourse.group/u/ydelafollye)\
**Post date:** [August 25, 2026, 12:35pm UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/29 "2026-08-25T12:35:47Z")

</div>

Hello,

We are seeing the same symptoms with a three-controller OpenZiti 2.0.1 cluster deployed using the official ziti-controller Helm chart 3.2.1. We use cert-manager 1.19.2 and trust-manager 0.24.0, following the PKI setup documented by the chart.

We use an Entra ID external JWT signer with JIT enrollment enabled for both certificate and token modes. We reproduce the issue when clients use User Session mode: authentication succeeds and the clients show their services as available, but no traffic goes through.  
The edge router logs:

`invalid client certificate for api session`

We followed the HA PKI layout documented by the official v3 Helm chart. Controller 1 creates the shared edge root, while each controller has its own intermediate edge signer issued by that root.

Controllers 2 and 3 explicitly reference controller 1’s root issuer:

```auto
edgeSignerPki:
  alternativeIssuer:
    name: ziti-controller1-edge-root-issuer
    kind: Issuer
    group: cert-manager.io

```

> To be fully transparent, I did not diagnose this issue or write the workaround myself. Codex identified the likely certificate-chain problem, proposed rebuilding the chains at controller startup, and generated the wrapper below.  
> I applied it to our environment and validated the results, but I only understand the general principle and not every internal detail of the router’s certificate validation.
> 
> So please correct the analysis if anything is wrong.

As I understand the cert-manager tls.crt files used by identity.cert and identity.server\_cert contain the controller leaf and its intermediate, but not the shared root. The shared root is available separately in the trust-manager bundle.

Cert-manager omits the root intentionally ( [Certificate resource - cert-manager Documentation](https://cert-manager.io/docs/usage/certificate/#target-secret) ), but in this particular OpenZiti path it seems that the router can end up validating against the intermediate associated with one controller instead of the common root.

That would explain why an API Session Certificate issued by one controller works, while a new one issued by another controller after reconnection or failover is rejected.

The generated workaround builds the following chain when each controller starts:

controller leaf → controller intermediate → shared root

It then starts the controller with a temporary configuration referencing the completed chains.

```auto
image:
  command:
    - /bin/bash
    - -ceu
  args:
    - |
      runtime_dir=/tmp/ziti-controller-runtime
      source_config=/etc/ziti/ziti-controller.yaml
      ca_bundle=/etc/ziti/ctrl-plane-cas/ctrl-plane-cas.crt
      client_cert=/etc/ziti/ctrl-plane-client-identity/tls.crt
      server_cert=/etc/ziti/ctrl-plane-identity/tls.crt

      for required_file in "${source_config}" "${ca_bundle}" "${client_cert}" "${server_cert}"; do
        if [[! -s "${required_file}"]]; then
          echo "required controller PKI file is missing or empty: ${required_file}" >&2
          exit 1
        fi
      done

      mkdir -p "${runtime_dir}"

      # The trust-manager bundle is ordered from issuer to trust anchor. Keep
      # its final certificate, which is the shared self-signed root.
      awk '
        /-----BEGIN CERTIFICATE-----/ { certificate = ""; in_certificate = 1 }
        in_certificate { certificate = certificate $0 ORS }
        /-----END CERTIFICATE-----/ { last_certificate = certificate; in_certificate = 0 }
        END { printf "%s", last_certificate }
      ' "${ca_bundle}" > "${runtime_dir}/shared-root.crt"

      root_subject="$(openssl x509 -in "${runtime_dir}/shared-root.crt" -noout -subject -nameopt RFC2253)"
      root_issuer="$(openssl x509 -in "${runtime_dir}/shared-root.crt" -noout -issuer -nameopt RFC2253)"

      if [["${root_subject#subject=}" != "${root_issuer#issuer=}"]]; then
        echo "the last certificate in ${ca_bundle} is not a self-signed root" >&2
        exit 1
      fi

      cat "${client_cert}" "${runtime_dir}/shared-root.crt" > "${runtime_dir}/client-chain.crt"
      cat "${server_cert}" "${runtime_dir}/shared-root.crt" > "${runtime_dir}/server-chain.crt"

      sed \
        -e "s|${client_cert}|${runtime_dir}/client-chain.crt|" \
        -e "s|${server_cert}|${runtime_dir}/server-chain.crt|" \
        "${source_config}" > "${runtime_dir}/ziti-controller.yaml"

      exec ziti controller run "${runtime_dir}/ziti-controller.yaml" "$@"

    # bash uses the first argument after the script as $0. The controller
    # chart appends the cluster agent flags after this placeholder.
    - ziti-controller-chain-wrapper

```

Since applying it, Windows and macOS User Sessions work fine, repeated tunnel reconnections continue to work, and connectivity still works after transferring leadership to another controller.

So, Does this look like the Helm chart manifestation of #4094 ( [Edge router rejects valid first-party client certificates when the edge signing CA differs from the ctrl-channel CA · Issue #4094 · openziti/ziti · GitHub](https://github.com/openziti/ziti/issues/4094) ), or is it still related to the shared-root case in #3857 ( [Router rejects API session certificate after HA failover for ext-jwt identities without enrollment · Issue #3857 · openziti/ziti · GitHub](https://github.com/openziti/ziti/issues/3857) )?

Is a fix planned, and would it be useful to open a separate issue in openziti/ziti or openziti/helm-charts?

Thanks

---

<div class="post-metadata">

**Author:** ![h.rouineb](https://avatars.discourse-cdn.com/v4/letter/h/53a042/32.png) [@h.rouineb](https://openziti.discourse.group/u/h.rouineb)\
**Post date:** [August 26, 2026, 9:02am UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/30 "2026-08-26T09:02:02Z")

</div>

Hi there,

We're facing the exact same issue. We are currently in a POC phase to evaluate whether to roll this out as the primary VPN across our entire corporation (GO/NO-GO decision).

While the core functionality works, the current developer experience isn't quite where we need it to be.

I noticed two PRs that seem to address this issue:

- [PR #4140](https://www.google.com/search?q=https://github.com/openziti/ziti/pull/4140&authuser=1) (accept first-party certs issued by a separate CA)

- [PR #4153](https://www.google.com/search?q=https://github.com/openziti/ziti/pull/4153&authuser=1) (backport to release-v2.0.x)

Is there any update or estimated timeline on when these fixes will be merged ?

---

<div class="post-metadata">

**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\
**Post date:** [August 26, 2026, 2:53pm UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/31 "2026-08-26T14:53:59Z")

</div>

I'm working on getting them merged now.

---

<div class="post-metadata">

**Author:** ![andrew.martinez](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/andrew.martinez/32/540_2.png) [@andrew.martinez](https://openziti.discourse.group/u/andrew.martinez)\
**Post date:** [August 26, 2026, 6:45pm UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/32 "2026-08-26T18:45:22Z")

</div>

Merged to latest and 2.0.x LTS.

Took longer than expected. An issue with another security patch would also have prevented multi-root-CA networks from working. Both PRs added integration tests for that setup.

---

<div class="post-metadata">

**Author:** ![h.rouineb](https://avatars.discourse-cdn.com/v4/letter/h/53a042/32.png) [@h.rouineb](https://openziti.discourse.group/u/h.rouineb)\
**Post date:** [August 27, 2026, 7:12am UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/33 "2026-08-27T07:12:55Z")

</div>

Thanks a lot for you hard work, I built locally the images/the binary, well try this ASAP, keep you informed.

---

<div class="post-metadata">

**Author:** ![h.rouineb](https://avatars.discourse-cdn.com/v4/letter/h/53a042/32.png) [@h.rouineb](https://openziti.discourse.group/u/h.rouineb)\
**Post date:** [August 28, 2026, 6:44am UTC](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812/34 "2026-08-28T06:44:40Z")

</div>

Indeed it works, thank you @andrew.martinez , I guess we'll have to wait for next LTS release (late september according to the release policy document).

[Previous page](https://openziti.discourse.group/t/ha-not-working-as-expected-with-ext-jwt-signer-without-jit-enrollment/5812.md?page=1)