# OpenZiti Helm Chart - Using Enterprise PKI Instead of Self-Signed CA

**URL:** <https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856>\
**Category:** Uncategorized\
**Created:** [January 31, 2025, 12:42pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856 "2025-01-31T12:42:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)\
**Post date:** [January 31, 2025, 12:42pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/1 "2025-01-31T12:42:02Z")

</div>

As I have understood after reviewing the **Helm chart YAML files** for `ziti-controller`, the default CA structure is as follows:

```auto
[ROOT CA] release-name-controller-selfsigned-ca-issuer (Self-Signed)
 ├── [ROOT CA] release-name-controller-ctrl-plane-root-issuer
 │ ├── [INTERMEDIATE CA] release-name-controller-ctrl-plane-intermediate-issuer
 │ │ ├── [CERT] release-name-controller-ctrl-plane-identity (Controller)
 │ │ ├── [CERT] release-name-controller-ctrl-plane-client-identity (Client)
 │
 ├── [ROOT CA] release-name-controller-edge-root-issuer
 │ ├── [INTERMEDIATE CA] release-name-controller-edge-signer-issuer
 │ ├── [CERT] release-name-controller-admin-client-cert (Admin)
 │
 ├── [ROOT CA] release-name-controller-web-root-issuer
 │ ├── [INTERMEDIATE CA] release-name-controller-web-intermediate-issuer
 │ ├── [CERT] release-name-controller-web-identity-cert (Web)
 │ ├── [CERT] release-name-controller-web-client-identity (Web Client)

```

* * *

### **Expected CA Structure with Enterprise PKI**

I want to use **Enterprise PKI as the root CA** instead of the Helm-generated self-signed CA, with the expected structure as follows:

```auto
[ROOT CA] Enterprise PKI Root CA <-- (Managed by Organization's Security Team)
 ├── [INTERMEDIATE CA] Enterprise PKI Intermediate CA <-- (Issued from Root CA)
 │ ├── [INTERMEDIATE CA] release-name-controller-ctrl-plane-root-issuer
 │ │ ├── [INTERMEDIATE CA] release-name-controller-ctrl-plane-intermediate-issuer
 │ │ │ ├── [CERT] release-name-controller-ctrl-plane-identity (Controller)
 │ │ │ ├── [CERT] release-name-controller-ctrl-plane-client-identity (Client)
 │ │
 │ ├── [INTERMEDIATE CA] release-name-controller-edge-root-issuer
 │ │ ├── [INTERMEDIATE CA] release-name-controller-edge-signer-issuer
 │ │ ├── [CERT] release-name-controller-admin-client-cert (Admin)
 │ │
 │ ├── [INTERMEDIATE CA] release-name-controller-web-root-issuer
 │ ├── [INTERMEDIATE CA] release-name-controller-web-intermediate-issuer
 │ │ ├── [CERT] release-name-controller-web-identity-cert (Web)
 │ │ ├── [CERT] release-name-controller-web-client-identity (Web Client)

```

* * *

### **What I Tried**

After reading the following sources:

- **OpenZiti Documentation:**
  - [Install the Controller in Kubernetes | NetFoundry Documentation](https://openziti.io/docs/guides/deployments/kubernetes/kubernetes-controller#advanced-pki)

- **Ziti-Controller Helm Chart Values YAML:**
  - [helm-charts/charts/ziti-controller/values.yaml at main · openziti/helm-charts · GitHub](https://github.com/openziti/helm-charts/blob/main/charts/ziti-controller/values.yaml)

I thought that **mentioning `ctrlPlane.alternativeIssuer.kind` & `ctrlPlane.alternativeIssuer.name`** in Helm values would allow Enterprise PKI to be used instead of the self-signed CA.

**However, the CA structure generated by Helm still uses self-signed issuers instead of the external PKI.**

* * *

### **What I Found**

After checking the **Helm templates (`templates/`) directory** in:  
🔗 [https://github.com/openziti/helm-charts/tree/main/charts/ziti-controller/templates](https://github.com/openziti/helm-charts/tree/main/charts/ziti-controller/templates)

I **do not see `ctrlPlane.alternativeIssuer.kind` & `ctrlPlane.alternativeIssuer.name` being used anywhere in the Helm templates**.

**This raises the question:**

- Does the Helm chart **support using an external CA** like Enterprise PKI?
- If yes, what values should be set in `values.yaml`?
- Or do we need to **manually create the CA and inject it via Kubernetes Secrets before Helm installation?**

* * *

### **`values.yaml` Used**

This is the `values.yaml` file we used:

```yaml
clientApi:
  advertisedHost: ziti-controller.example.com
  service:
    enabled: true
    type: ClusterIP

  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      external-dns.alpha.kubernetes.io/hostname: "ziti-controller.example.com"
      service.beta.kubernetes.io/aws-load-balancer-internal: "false" # Ensures the LB is public
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0d5ee2792bcbd86a4"
      service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"

ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP
  alternativeIssuer:
    kind: ClusterIssuer
    name: enterprise-pki-cluster-issuer

highAvailability:
  mode: standalone
  replicas: 1

persistence:
  enabled: true
  storageClass: "ebs-sc"
  accessMode: ReadWriteOnce
  size: 3Gi

cert-manager:
  enabled: false # Installed it manually in ziti-controller namespace

trust-manager:
  enabled: false # Installed it manually in ziti-controller namespace

ingress-nginx:
  enabled: true
  controller:
    extraArgs:
      enable-ssl-passthrough: "true"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0d5ee2792bcbd86a4"
        service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"

```

Despite setting `ctrlPlane.alternativeIssuer.kind=ClusterIssuer` and `ctrlPlane.alternativeIssuer.name=enterprise-pki-cluster-issuer`, the Helm deployment **still created self-signed CAs instead of using the external PKI**.

* * *

### **Request for Help**

1. **How can we correctly use Enterprise PKI as the root CA?**
2. **Is there a proper way to override the Helm chart defaults to prevent self-signed CA creation?**
3. **Do we need to manually provision the CA secrets and then reference them in the Helm values?**
4. **Has anyone successfully used an external CA with OpenZiti in Kubernetes?**

Any help or guidance would be much appreciated!

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [January 31, 2025, 4:41pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/2 "2025-01-31T16:41:15Z")

</div>

> [@am3y](#):
>
> 1. **How can we correctly use Enterprise PKI as the root CA?**
> 2. **Is there a proper way to override the Helm chart defaults to prevent self-signed CA creation?**
> 3. **Do we need to manually provision the CA secrets and then reference them in the Helm values?**
> 4. **Has anyone successfully used an external CA with OpenZiti in Kubernetes?**

You're on the right track, @am3y. To clarify for all readers, you're looking to change the root of trust for the PKI that issues the Ziti controller's identity, not to configure the Ziti controller to trust a 3rd party CA for enrolling identities.

You can accomplish this enterprise single-root-of-trust model by deploying a new Ziti controller with these settings:

1. set the alternative issuer to the Cert Manager issuer ref you wish to issue the controller's identity ([link to Helm template where `.ctrlPlane.alternativeIssuer` is used](https://github.com/openziti/helm-charts/blob/ziti-controller-1.1.17/charts/ziti-controller/templates/ca-router-ctrl-identity.yaml#L65)).
2. disable the independent web PKI, causing the controller to have only one PKI from your specified root: `webBindingPki.enabled=false`.

Known issues that can be fixed in future releases:

- some unnecessary PKI resources are created and unused with this approach
- some web PKI features are unavailable when web PKI is disabled, e.g., alt server certs for the web console

Clarifying information: `{{ .Release.Name }}-controller-edge-root-issuer` will continue to be created and is used to issue the intermediate CA cert that is the edge signer CA from which router and identity enrollments are created because the Ziti controller must own a CA for this purpose.

The controller also supports adding 3rd party CAs (CAs that are external to it's own edge signer CA) for identity enrollment, e.g. SPIRE.

There's no reason we couldn't change the chart so this intermediate CA that is the edge signer CA has an alternative issuer like the ctrl plane CA. In that case, Ziti's edge signer CA has the same root of trust as the controller's identity.

---

<div class="post-metadata">

**Author:** ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)\
**Post date:** [February 2, 2025, 1:35pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/3 "2025-02-02T13:35:03Z")

</div>

Dear @qrkourier

Thank you for your prompt and insightful response to my query. I appreciate your guidance and wanted to let you know that I have successfully addressed the PKI configuration for the Ziti controller.

Now, I’m focusing on setting up the Ziti router and configuring identities and services. Are there any specific considerations or configurations I should be aware of to ensure everything integrates properly with my Enterprise PKI setup?

Looking forward to your insights!

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [February 2, 2025, 6:34pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/4 "2025-02-02T18:34:48Z")

</div>

You're welcome. I'm glad you found an acceptable solution.

I don't know if the following information will be important for your case, so I'll just state some facts in case they're helpful.

The Ziti controller manages the identity for all routers. The routers certificates are automatically issued by the built-in edge signer CA. When a router runs with the `--extend` option, then it will immediately renew its identity certificates. This will occur automatically, periodically as well, assuming the filesystem is writable where the identity files are located. The router chart provides a PVC for this purpose.

You have two categories of enrollments you can use to recognize a Ziti (endpoint) identity, e.g. for an edge tunneler: enroll with a certificate from the built-in edge signer CA (default option) or enroll with a certificate from another CA you control and have administratively verified in the Ziti controller.

If you choose to verify your other CA, then you may use it to enroll identities with certificates you issued independently (one ziti identity per certificate), or you may configure the Ziti controller to recognize a "claim" in the certificate, e.g. spiffe id, to represent an existing Ziti identity (many short lived certificates with identical claim map to one Ziti identity by external ID).

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [March 26, 2025, 7:20pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/5 "2025-03-26T19:20:29Z")

</div>

Hi all,

I am working on a similar setup, where I would like to use a private CA to manage the certificates. Currently I already have cert-manager deployed on a cluster with a custom ClusterIssuer.

I followed the same values.yml file as am3y, and added my alternativeIssuer.kind & name.

I am not sure if this is a bug, or a misconfig somewhere, but it seems a CSR is being created every few seconds (When I run get certificaterequest -n , i see what seems like hundreds of CSR objects, however when I run get certificate I see 1 seemingly valid/READY certificate from my private issuer and 2 not ready from the openziti-controller-ctrl-plane-intermediate-issuer instead of the private CA issuer)

Here is my values.yaml file

```auto
clientApi:
  containerPort: 1280
  advertisedHost: "<redacted>"
  advertisedPort: 443
  service:
    enabled: true
  ingress:
    enabled: true
    ingressClassName: "nginx"
    labels: {}
    annotations: {
      kubernetes.io/ingress.allow-http: "false",
      nginx.ingress.kubernetes.io/ssl-passthrough: "true",
      <redacted>,
    }

ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP
  alternativeIssuer: 
    kind: ClusterIssuer
    name: <redacted>

cert-manager:
  enabled: false

trust-manager:
  enabled: false

```

TLDR: with the setup above, should I be seeing a CSR being created every few seconds, and 2 Certs in False Ready state with a different issuer then the ready one from my CA?

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [March 26, 2025, 7:41pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/6 "2025-03-26T19:41:38Z")

</div>

Hey there, welcome! 👋

Which chart version are you working with?

e.g.,

```bash
helm list -n ziti

```

Here's how I expect it to work, assuming you want all three of Ziti's intermediate issuers to descend from a single ClusterIssuer. This will clarify why certificates were requested from `openziti-controller-ctrl-plane-intermediate-issuer` when the alternative issuer was specified. In short, leaves will still be requested from Ziti's three intermediate issuers, and the alternative issuer option causes any or all of Ziti's three intermediate issuers to themselves be issued by your alternative issuer.

In your input values YAML, specify the same alternative issuer for each of Ziti's three PKIs, representing the three intermediate issuers it will request from your specified ClusterIssuer.

1. `ctrlPlane.alternativeIssuer` - during deployment, ctrlPlane intermediate issues the controller's identity leaves for controller-to-controller and router-to-controller "control plane" protocol
2. `webBindingPki.alternativeIssuer` - during deployment, the web intermediate issues leaves for the controller's web listeners where HTTP APIs, the console, etc. are served
3. `edgeSignerPki.alternativeIssuer` - edge signer intermediate issues edge identities' leaves during enrollment and is managed internally by the Ziti controller

The latest chart should not provision any root issuers in this scenario, and all three of Ziti's intermediate issuers should descend from the alternative issuer you specified.

from: [Install the Controller in Kubernetes | OpenZiti](https://openziti.io/docs/guides/deployments/kubernetes/kubernetes-controller)

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [March 26, 2025, 8:26pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/7 "2025-03-26T20:26:30Z")

</div>

Thanks for the quick reply!

I am on chart v 1.2.2 for the controller.

I have the alternative Issuers set for all 3 intermediate issuers. I am still seeing a loop of CSRs being created.

Here is my output for get certificate -n openziti

```auto
NAME READY SECRET AGE
openziti-dev-controller-ctrl-plane-client-identity False openziti-dev-controller-ctrl-plane-client-identity-secret 49s
openziti-dev-controller-ctrl-plane-identity False openziti-dev-controller-ctrl-plane-identity-secret 49s
openziti-dev-controller-ctrl-plane-intermediate-cert True openziti-dev-controller-ctrl-plane-intermediate-secret 49s
openziti-dev-controller-edge-signer-cert True openziti-dev-controller-edge-signer-secret 49s
openziti-dev-controller-web-client-identity False openziti-dev-controller-web-client-identity-secret 49s
openziti-dev-controller-web-identity-cert False openziti-dev-controller-web-identity-secret 49s
openziti-dev-controller-web-intermediate-cert True openziti-dev-controller-web-intermediate-secret 49s
trust-manager True trust-manager-tls 49s

```

I wont paste the entire get certificaterequest output, but after a few minutes we have 50+ different CSRs with new ones being created every couple minutes or so. All my CSRs using my top level private CA have approved true && ready true

```auto
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
openziti-dev-controller-ctrl-plane-client-identity-qkjzr True False openziti-dev-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 3m8s
openziti-dev-controller-ctrl-plane-identity-6tnbd True False openziti-dev-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 3m2s
openziti-dev-controller-ctrl-plane-intermediate-cert-24d9p True True <redacted> system:serviceaccount:cert-manager:cert-manager 2m49s
openziti-dev-controller-ctrl-plane-intermediate-cert-2zfm2 True True <redacted> system:serviceaccount:cert-manager:cert-manager 32s
openziti-dev-controller-web-client-identity-zd65w True False openziti-dev-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 3m7s
openziti-dev-controller-web-identity-cert-gj5sn True False openziti-dev-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 3m3s
openziti-dev-controller-web-intermediate-cert-2967t True True <redacted> system:serviceaccount:cert-manager:cert-manager 68s
openziti-dev-controller-web-intermediate-cert-44jrk True True <redacted> system:serviceaccount:cert-manager:cert-manager 84s
openziti-dev-controller-edge-signer-cert-2xc2v True True <redacted> system:serviceaccount:cert-manager:cert-manager 35s
openziti-dev-controller-edge-signer-cert-48tgv True True <redacted> system:serviceaccount:cert-manager:cert-manager 64s
trust-manager-gmdxt True True trust-manager system:serviceaccount:cert-manager:cert-manager 3m6s

```

Perhaps it could be an issue with the PKI tool I am using (hashicorp vault) and the role policies attached to the issuer, as I had to make a role with specific key and name permissions. Maybe I need to check the config if it is permitted to issue intermediate certificates.

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [March 26, 2025, 9:08pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/8 "2025-03-26T21:08:11Z")

</div>

The distribution of failures is concentrated on the leaves requested from Ziti's intermediates, whereas Ziti's intermediates that were requested from your alternative issuer succeeded. I should be able to reproduce this if it's not limited to Vault-managed issuers.

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [March 27, 2025, 1:56pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/9 "2025-03-27T13:56:57Z")

</div>

For reference, here's a healthy list of CSRs in the ziti namespace following a successful controller deployment with default PKI, not using the alternative issuer.

```bash
❯ k -n miniziti get certificaterequests.cert-manager.io
NAME APPROVED DENIED READY ISSUER REQUESTER AGE
ziti-controller-ctrl-plane-client-identity-1 True True ziti-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-ctrl-plane-identity-1 True True ziti-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-ctrl-plane-intermediate-cert-1 True True ziti-controller-ctrl-plane-root-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-ctrl-plane-root-cert-1 True True ziti-controller-selfsigned-ca-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-edge-root-cert-1 True True ziti-controller-selfsigned-ca-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-edge-signer-cert-1 True True ziti-controller-edge-root-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-web-client-identity-1 True True ziti-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-web-identity-cert-1 True True ziti-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-web-intermediate-cert-1 True True ziti-controller-web-root-issuer system:serviceaccount:cert-manager:cert-manager 22h
ziti-controller-web-root-cert-1 True True ziti-controller-selfsigned-ca-issuer system:serviceaccount:cert-manager:cert-manager 22h

```

I've started working to reproduce it and haven't yet seen the flurry of CSRs, but I do not see 100% expected behavior with alternative issuers. I tried setting edge and web to use ctrlPlane root CA, and still the web cert is issued by the old web root. I will continue exploring later today or tomorrow. There may be an issue with CSRs or trust bundle or both.

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [March 31, 2025, 3:45pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/10 "2025-03-31T15:45:33Z")

</div>

Good morning!

Thanks for looking into this! Today I'll test with the default PKI (no alt issuer) however It would be nice to not have to add another CA to a root store on endpoints.

Just another bit of info for you, the behavior was the same regardless of if I set trust-manager && cert-manager to "enabled: true" or "enabled: false".

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [March 31, 2025, 3:55pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/11 "2025-03-31T15:55:00Z")

</div>

The CM/TM "enabled" input values control whether or not those Helm subcharts (dependencies) are also deployed when you deploy the ziti-controller umbrella (parent) chart.

Working with a single PKI, the `ctrlPlane` root CA, I still didn't encounter a runaway flurry of CSRs, but I noticed that setting alternative issuers can create a deadlock, and addressed that specific problem in [Release ziti-controller-1.3.0 · openziti/helm-charts · GitHub](https://github.com/openziti/helm-charts/releases/tag/ziti-controller-1.3.0)

Let's diagnose precisely what's going wrong by referencing this overview of the Cert Manager workflow. A crucial error event is probably on one of CM's custom resources.

In this case, we're only dealing with a custom root, not a public root like LetsEncrypt, so there will be no Orders, Challenges custom resources in play. I think the workflow is like this:

1. Create (Cluster)Issuer
2. create Certificate referencing the (Cluster)Issuer
3. cert Controller creates CertificateRequest, creates an opaque Secret w/ private key in data key `tls.key`, waits for issuance to update Secret with `tls.crt`, `ca.crt` (check cert-manager pod logs for clues to this async process)

Does that match your understanding?

Does tracing the failure this way reveal any new clues?

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [March 31, 2025, 6:44pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/12 "2025-03-31T18:44:11Z")

</div>

There's quite a bit of logs, mainly it looks like the custom issuer that the templates create are not found, I confirmed this by running

Here are some examples of those logs

```auto
cert-manager/issuers "msg"="issuer in work queue no longer exists" "error"="issuer.cert-manager.io \"openziti-dev-controller-edge-signer-issuer\" not found"
cert-manager/certificaterequests-issuer-venafi "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-web-identity-cert-vwt55\": the object has been modified; please apply your changes to the latest version and try again" "key"="openziti/openziti-dev-controller-web-identity-cert-vwt55"
cert-manager/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-web-client-identity-bk2w8\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/openziti/openziti-dev-controller-web-client-identity-bk2w8, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 58357f48-4360-40ca-9337-18c30be4383a, UID in object meta: " "key"="openziti/openziti-dev-controller-web-client-identity-bk2w8"
cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-web-identity-cert-vwt55\": the object has been modified; please apply your changes to the latest version and try again" "key"="openziti/openziti-dev-controller-web-identity-cert-vwt55"
cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"openziti-dev-controller-web-intermediate-cert\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificates/openziti/openziti-dev-controller-web-intermediate-cert, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 391ecadb-1a68-45ef-aa7a-164c3128b8d9, UID in object meta: " "key"="openziti/openziti-dev-controller-web-intermediate-cert"
cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"openziti-dev-controller-edge-signer-cert\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificates/openziti/openziti-dev-controller-edge-signer-cert, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 43566531-f724-4e45-8c06-ffb33bc37095, UID in object meta: " "key"="openziti/openziti-dev-controller-edge-signer-cert"
cert-manager/certificaterequests-issuer-venafi "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-ctrl-plane-client-identity-dzzvv\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/openziti/openziti-dev-controller-ctrl-plane-client-identity-dzzvv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 54b86ce6-85cc-470f-873f-fe207d275687, UID in object meta: " "key"="openziti/openziti-dev-controller-ctrl-plane-client-identity-dzzvv"

```

When running the helm template without alt issuers, all my CSRs are set to Ready True and there is no looping

```auto
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
openziti-dev-controller-ctrl-plane-client-identity-rrbc7 True True openziti-dev-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 30s
openziti-dev-controller-ctrl-plane-identity-btjbb True True openziti-dev-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 28s
openziti-dev-controller-ctrl-plane-intermediate-cert-6mrkt True True openziti-dev-controller-ctrl-plane-root-issuer system:serviceaccount:cert-manager:cert-manager 30s
openziti-dev-controller-ctrl-plane-root-cert-f6l95 True True openziti-dev-controller-selfsigned-ca-issuer system:serviceaccount:cert-manager:cert-manager 32s
openziti-dev-controller-edge-root-cert-cpbsf True True openziti-dev-controller-selfsigned-ca-issuer system:serviceaccount:cert-manager:cert-manager 33s
openziti-dev-controller-edge-signer-cert-j278c True True openziti-dev-controller-edge-root-issuer system:serviceaccount:cert-manager:cert-manager 31s
openziti-dev-controller-web-client-identity-ls6pk True True openziti-dev-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 32s
openziti-dev-controller-web-identity-cert-m6mtr True True openziti-dev-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 27s
openziti-dev-controller-web-intermediate-cert-k25lv True True openziti-dev-controller-web-root-issuer system:serviceaccount:cert-manager:cert-manager 32s
openziti-dev-controller-web-root-cert-4qq5d True True openziti-dev-controller-selfsigned-ca-issuer system:serviceaccount:cert-manager:cert-manager 32s

```

however now I am running into another issue as the openziti controller pod is stuck in init state

```auto
 Warning FailedMount 4m35s (x5 over 4m43s) kubelet MountVolume.SetUp failed for volume "ziti-controller-ctrl-plane-cas" : configmap "openziti-dev-controller-ctrl-plane-cas" not found
  Warning FailedMount 4m35s (x5 over 4m43s) kubelet MountVolume.SetUp failed for volume "cert-ctrl-plane-identity" : secret "openziti-dev-controller-ctrl-plane-identity-secret" not found
  Warning FailedMount 4m35s (x5 over 4m43s) kubelet MountVolume.SetUp failed for volume "cert-web-client-identity" : secret "openziti-dev-controller-web-client-identity-secret" not found
  Warning FailedMount 4m35s (x5 over 4m43s) kubelet MountVolume.SetUp failed for volume "cert-ctrl-plane-client-identity" : secret "openziti-dev-controller-ctrl-plane-client-identity-secret" not found
  Warning FailedMount 4m35s (x5 over 4m43s) kubelet MountVolume.SetUp failed for volume "cert-web-identity" : secret "openziti-dev-controller-web-identity-secret" not found

```

I confirmed the secrets its trying to mount exist

```auto
openziti-dev-controller-admin-secret Opaque 2 2m39s
openziti-dev-controller-ctrl-plane-client-identity-secret kubernetes.io/tls 3 92s
openziti-dev-controller-ctrl-plane-identity-secret kubernetes.io/tls 3 92s
openziti-dev-controller-ctrl-plane-intermediate-secret kubernetes.io/tls 3 2m13s
openziti-dev-controller-ctrl-plane-root-secret kubernetes.io/tls 3 2m32s
openziti-dev-controller-edge-root-secret kubernetes.io/tls 3 2m33s
openziti-dev-controller-edge-signer-secret kubernetes.io/tls 3 2m30s
openziti-dev-controller-trust-domain Opaque 1 2m39s
openziti-dev-controller-web-client-identity-secret kubernetes.io/tls 3 92s
openziti-dev-controller-web-identity-secret kubernetes.io/tls 3 92s
openziti-dev-controller-web-intermediate-secret kubernetes.io/tls 3 2m13s
openziti-dev-controller-web-root-secret kubernetes.io/tls 3 2m32s

```

I'll try with the new release of 1.3.0 and report back!

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [March 31, 2025, 7:06pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/13 "2025-03-31T19:06:02Z")

</div>

I'm still experiencing the issue with your controller chart 1.3.0.

I'm sure I have a config issue somewhere, as you couldn't reproduce the issue.

Thanks again for your help and I'll report back if I find a solution.

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [March 31, 2025, 10:53pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/14 "2025-03-31T22:53:54Z")

</div>

I wasn't able to reproduce with an alternative ClusterIssuer. Here's what my inputs looked like for the successful test.

I started with a fresh instance of K8s v1.35 w/ latest CM/TM.

I created a ClusterIssuer named "new-root-clusterissuer" to serve as my single root CA for Ziti's three intermediates.

```yaml
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: new-root-selfsigned
spec:
  selfSigned: {}

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: new-root-clusterissuer
spec:
  ca:
    secretName: new-root-cert-secret

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: new-root-cert
  namespace: cert-manager
spec:
  commonName: new-root-cert
  secretName: new-root-cert-secret
  isCA: true
  duration: 87840h
  renewBefore: 720h
  usages:
    - digital signature
    - cert sign
    - crl sign
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    kind: ClusterIssuer
    name: new-root-selfsigned

```

I deployed the Ziti controller with subcharts disabled since I already installed CM/TM. I used input values like these to base Ziti's intermediates on my alternative root, the same root for all three in this case.

```yaml
ctrlPlane:
  alternativeIssuer:
    kind: ClusterIssuer
    name: new-root-clusterissuer
edgeSignerPki:
  alternativeIssuer:
    kind: ClusterIssuer
    name: new-root-clusterissuer
webBindingPki:
  alternativeIssuer:
    kind: ClusterIssuer
    name: new-root-clusterissuer

```

```bash
❯ k get certificaterequests.cert-manager.io
NAME APPROVED DENIED READY ISSUER REQUESTER AGE
ziti-controller-ctrl-plane-client-identity-1 True True ziti-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 171m
ziti-controller-ctrl-plane-identity-1 True True ziti-controller-ctrl-plane-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 171m
ziti-controller-ctrl-plane-intermediate-cert-1 True True new-root-clusterissuer system:serviceaccount:cert-manager:cert-manager 171m
ziti-controller-edge-signer-cert-1 True True new-root-clusterissuer system:serviceaccount:cert-manager:cert-manager 171m
ziti-controller-web-client-identity-1 True True ziti-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 171m
ziti-controller-web-identity-cert-1 True True ziti-controller-web-intermediate-issuer system:serviceaccount:cert-manager:cert-manager 171m
ziti-controller-web-intermediate-cert-1 True True new-root-clusterissuer system:serviceaccount:cert-manager:cert-manager 171m

```

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [April 4, 2025, 8:56pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/15 "2025-04-04T20:56:23Z")

</div>

Coincidentally, Vault was mentioned again to me and now I'm curious about their integration with Cert Manager.

Are you using CM's Issuer spec like this to delegate issuance to Vault?

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: one-root-to-rule-them-all
spec:
  vault:
    path: pki_int/sign/example-dot-com
    server: https://vault.local
    caBundle: <base64 encoded CA Bundle PEM file>
    auth:
      ...

```

ref: [Vault - cert-manager Documentation](https://cert-manager.io/docs/configuration/vault/)

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [April 9, 2025, 4:53pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/16 "2025-04-09T16:53:58Z")

</div>

Hey!

Yea my cluster issuer looks like this

```auto
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: <Issuer-Name>
spec:
  vault:
    auth:
      appRole:
        path: approle
        roleId: <redacted>
        secretRef:
          key: <redacted>
          name: <redacted>
    caBundle: <base64 bundle pem file>
    path: pki_int/sign/example-dot-com
    server: https://vault-server.domain

```

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [April 9, 2025, 5:18pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/17 "2025-04-09T17:18:35Z")

</div>

Thank you for confirming that. I will eventually try to reproduce this with a Hashicorp Vault delegated issuer. It'd be good to know if we need to do something in particular to support their integration with Cert Manager. I was not able to reproduce the issue that you encountered by using another non-Vault issuer.

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [April 9, 2025, 5:25pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/18 "2025-04-09T17:25:15Z")

</div>

Thank you for looking into it! I did some experimentation last week without much luck. I think the good news is the issue lies in Vault so if I find any solution I will report back.

---

<div class="post-metadata">

**Author:** ![Tetrusp](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/tetrusp/32/3329_2.png) [@Tetrusp](https://openziti.discourse.group/u/Tetrusp)\
**Post date:** [April 9, 2025, 5:43pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/19 "2025-04-09T17:43:29Z")

</div>

I forgot to mention what I had stumbled upon last week.

```auto
cert-manager/issuers/setup "msg"="signing certificate is not a CA" "error"=null "related_resource_kind"="Secret" "related_resource_name"="openziti-dev-controller-web-intermediate-secret" "related_resource_namespace"="openziti" "resource_kind"="Issuer" "resource_name"="openziti-dev-controller-web-intermediate-issuer" "resource_namespace"="openziti" "resource_version"="v1"

```

Describe on that issuer

```auto
Spec:
  Ca:
    Secret Name: openziti-dev-controller-web-intermediate-secret
Status:
  Conditions:
    Message: Error getting keypair for CA issuer: certificate is not a CA

```

Above is one of the errors I was getting, which leads me to figure out why vault is not issuing a CA cert to the custom issuers listed out in the values file. Most likely vault is configured not to issue out CA certs to cert-manager, but I need to figure out why. I made a brand new clusterissuer from scratch with no restrictions to test and I got the same issue.

(I took a peek at the secret/certificate it issued, and it was indeed missing the CA flag which is odd because the INT ca for that issuer does have subject type CA)

I think the cluster-dot-com role I am using is missing something personally.

---

<div class="post-metadata">

**Author:** ![qrkourier](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/qrkourier/32/52_2.png) [@qrkourier](https://openziti.discourse.group/u/qrkourier)\
**Post date:** [April 9, 2025, 5:52pm UTC](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856/20 "2025-04-09T17:52:27Z")

</div>

That's a promising lead!

[Next page](https://openziti.discourse.group/t/openziti-helm-chart-using-enterprise-pki-instead-of-self-signed-ca/3856.md?page=2)