OpenZiti Helm Chart - Using Enterprise PKI Instead of Self-Signed CA

As I have understood after reviewing the Helm chart YAML files for ziti-controller, the default CA structure is as follows:

[ROOT CA] release-name-controller-selfsigned-ca-issuer (Self-Signed)
 ├── [ROOT CA] release-name-controller-ctrl-plane-root-issuer
 │    ├── [INTERMEDIATE CA] release-name-controller-ctrl-plane-intermediate-issuer
 │    │    ├── [CERT] release-name-controller-ctrl-plane-identity (Controller)
 │    │    ├── [CERT] release-name-controller-ctrl-plane-client-identity (Client)
 │
 ├── [ROOT CA] release-name-controller-edge-root-issuer
 │    ├── [INTERMEDIATE CA] release-name-controller-edge-signer-issuer
 │         ├── [CERT] release-name-controller-admin-client-cert (Admin)
 │
 ├── [ROOT CA] release-name-controller-web-root-issuer
 │    ├── [INTERMEDIATE CA] release-name-controller-web-intermediate-issuer
 │         ├── [CERT] release-name-controller-web-identity-cert (Web)
 │         ├── [CERT] release-name-controller-web-client-identity (Web Client)

Expected CA Structure with Enterprise PKI

I want to use Enterprise PKI as the root CA instead of the Helm-generated self-signed CA, with the expected structure as follows:

[ROOT CA] Enterprise PKI Root CA  <-- (Managed by Organization's Security Team)
 ├── [INTERMEDIATE CA] Enterprise PKI Intermediate CA  <-- (Issued from Root CA)
 │    ├── [INTERMEDIATE CA] release-name-controller-ctrl-plane-root-issuer
 │    │    ├── [INTERMEDIATE CA] release-name-controller-ctrl-plane-intermediate-issuer
 │    │    │    ├── [CERT] release-name-controller-ctrl-plane-identity (Controller)
 │    │    │    ├── [CERT] release-name-controller-ctrl-plane-client-identity (Client)
 │    │
 │    ├── [INTERMEDIATE CA] release-name-controller-edge-root-issuer
 │    │    ├── [INTERMEDIATE CA] release-name-controller-edge-signer-issuer
 │    │         ├── [CERT] release-name-controller-admin-client-cert (Admin)
 │    │
 │    ├── [INTERMEDIATE CA] release-name-controller-web-root-issuer
 │         ├── [INTERMEDIATE CA] release-name-controller-web-intermediate-issuer
 │         │    ├── [CERT] release-name-controller-web-identity-cert (Web)
 │         │    ├── [CERT] release-name-controller-web-client-identity (Web Client)

What I Tried

After reading the following sources:

I thought that mentioning ctrlPlane.alternativeIssuer.kind & ctrlPlane.alternativeIssuer.name in Helm values would allow Enterprise PKI to be used instead of the self-signed CA.

However, the CA structure generated by Helm still uses self-signed issuers instead of the external PKI.


What I Found

After checking the Helm templates (templates/) directory in:
:link: https://github.com/openziti/helm-charts/tree/main/charts/ziti-controller/templates

I do not see ctrlPlane.alternativeIssuer.kind & ctrlPlane.alternativeIssuer.name being used anywhere in the Helm templates.

This raises the question:

  • Does the Helm chart support using an external CA like Enterprise PKI?
  • If yes, what values should be set in values.yaml?
  • Or do we need to manually create the CA and inject it via Kubernetes Secrets before Helm installation?

values.yaml Used

This is the values.yaml file we used:

clientApi:
  advertisedHost: ziti-controller.example.com
  service:
    enabled: true
    type: ClusterIP

  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      external-dns.alpha.kubernetes.io/hostname: "ziti-controller.example.com"
      service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # Ensures the LB is public
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0d5ee2792bcbd86a4"
      service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"

ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP
  alternativeIssuer:
    kind: ClusterIssuer
    name: enterprise-pki-cluster-issuer

highAvailability:
  mode: standalone
  replicas: 1

persistence:
  enabled: true
  storageClass: "ebs-sc"
  accessMode: ReadWriteOnce
  size: 3Gi

cert-manager:
  enabled: false # Installed it manually in ziti-controller namespace

trust-manager:
  enabled: false # Installed it manually in ziti-controller namespace

ingress-nginx:
  enabled: true
  controller:
    extraArgs:
      enable-ssl-passthrough: "true"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0d5ee2792bcbd86a4"
        service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"

Despite setting ctrlPlane.alternativeIssuer.kind=ClusterIssuer and ctrlPlane.alternativeIssuer.name=enterprise-pki-cluster-issuer, the Helm deployment still created self-signed CAs instead of using the external PKI.


Request for Help

  1. How can we correctly use Enterprise PKI as the root CA?
  2. Is there a proper way to override the Helm chart defaults to prevent self-signed CA creation?
  3. Do we need to manually provision the CA secrets and then reference them in the Helm values?
  4. Has anyone successfully used an external CA with OpenZiti in Kubernetes?

Any help or guidance would be much appreciated!

1 Like

You're on the right track, @am3y. To clarify for all readers, you're looking to change the root of trust for the PKI that issues the Ziti controller's identity, not to configure the Ziti controller to trust a 3rd party CA for enrolling identities.

You can accomplish this enterprise single-root-of-trust model by deploying a new Ziti controller with these settings:

  1. set the alternative issuer to the Cert Manager issuer ref you wish to issue the controller's identity (link to Helm template where .ctrlPlane.alternativeIssuer is used).
  2. disable the independent web PKI, causing the controller to have only one PKI from your specified root: webBindingPki.enabled=false.

Known issues that can be fixed in future releases:

  • some unnecessary PKI resources are created and unused with this approach
  • some web PKI features are unavailable when web PKI is disabled, e.g., alt server certs for the web console

Clarifying information: {{ .Release.Name }}-controller-edge-root-issuer will continue to be created and is used to issue the intermediate CA cert that is the edge signer CA from which router and identity enrollments are created because the Ziti controller must own a CA for this purpose.

The controller also supports adding 3rd party CAs (CAs that are external to it's own edge signer CA) for identity enrollment, e.g. SPIRE.

There's no reason we couldn't change the chart so this intermediate CA that is the edge signer CA has an alternative issuer like the ctrl plane CA. In that case, Ziti's edge signer CA has the same root of trust as the controller's identity.

1 Like

Dear @qrkourier

Thank you for your prompt and insightful response to my query. I appreciate your guidance and wanted to let you know that I have successfully addressed the PKI configuration for the Ziti controller.

Now, I’m focusing on setting up the Ziti router and configuring identities and services. Are there any specific considerations or configurations I should be aware of to ensure everything integrates properly with my Enterprise PKI setup?

Looking forward to your insights!

1 Like

You're welcome. I'm glad you found an acceptable solution.

I don't know if the following information will be important for your case, so I'll just state some facts in case they're helpful.

The Ziti controller manages the identity for all routers. The routers certificates are automatically issued by the built-in edge signer CA. When a router runs with the --extend option, then it will immediately renew its identity certificates. This will occur automatically, periodically as well, assuming the filesystem is writable where the identity files are located. The router chart provides a PVC for this purpose.

You have two categories of enrollments you can use to recognize a Ziti (endpoint) identity, e.g. for an edge tunneler: enroll with a certificate from the built-in edge signer CA (default option) or enroll with a certificate from another CA you control and have administratively verified in the Ziti controller.

If you choose to verify your other CA, then you may use it to enroll identities with certificates you issued independently (one ziti identity per certificate), or you may configure the Ziti controller to recognize a "claim" in the certificate, e.g. spiffe id, to represent an existing Ziti identity (many short lived certificates with identical claim map to one Ziti identity by external ID).

Hi all,

I am working on a similar setup, where I would like to use a private CA to manage the certificates. Currently I already have cert-manager deployed on a cluster with a custom ClusterIssuer.

I followed the same values.yml file as am3y, and added my alternativeIssuer.kind & name.

I am not sure if this is a bug, or a misconfig somewhere, but it seems a CSR is being created every few seconds (When I run get certificaterequest -n , i see what seems like hundreds of CSR objects, however when I run get certificate I see 1 seemingly valid/READY certificate from my private issuer and 2 not ready from the openziti-controller-ctrl-plane-intermediate-issuer instead of the private CA issuer)

Here is my values.yaml file

clientApi:
  containerPort: 1280
  advertisedHost: "<redacted>"
  advertisedPort: 443
  service:
    enabled: true
  ingress:
    enabled: true
    ingressClassName: "nginx"
    labels: {}
    annotations: {
      kubernetes.io/ingress.allow-http: "false",
      nginx.ingress.kubernetes.io/ssl-passthrough: "true",
      <redacted>,
    }

ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP
  alternativeIssuer: 
    kind: ClusterIssuer
    name: <redacted>

cert-manager:
  enabled: false

trust-manager:
  enabled: false

TLDR: with the setup above, should I be seeing a CSR being created every few seconds, and 2 Certs in False Ready state with a different issuer then the ready one from my CA?

Hey there, welcome! :waving_hand:

Which chart version are you working with?

e.g.,

helm list -n ziti

Here's how I expect it to work, assuming you want all three of Ziti's intermediate issuers to descend from a single ClusterIssuer. This will clarify why certificates were requested from openziti-controller-ctrl-plane-intermediate-issuer when the alternative issuer was specified. In short, leaves will still be requested from Ziti's three intermediate issuers, and the alternative issuer option causes any or all of Ziti's three intermediate issuers to themselves be issued by your alternative issuer.

In your input values YAML, specify the same alternative issuer for each of Ziti's three PKIs, representing the three intermediate issuers it will request from your specified ClusterIssuer.

  1. ctrlPlane.alternativeIssuer - during deployment, ctrlPlane intermediate issues the controller's identity leaves for controller-to-controller and router-to-controller "control plane" protocol
  2. webBindingPki.alternativeIssuer - during deployment, the web intermediate issues leaves for the controller's web listeners where HTTP APIs, the console, etc. are served
  3. edgeSignerPki.alternativeIssuer - edge signer intermediate issues edge identities' leaves during enrollment and is managed internally by the Ziti controller

The latest chart should not provision any root issuers in this scenario, and all three of Ziti's intermediate issuers should descend from the alternative issuer you specified.

from: Install the Controller in Kubernetes | OpenZiti

Thanks for the quick reply!

I am on chart v 1.2.2 for the controller.

I have the alternative Issuers set for all 3 intermediate issuers. I am still seeing a loop of CSRs being created.

Here is my output for get certificate -n openziti

NAME                                                   READY   SECRET                                                      AGE
openziti-dev-controller-ctrl-plane-client-identity     False   openziti-dev-controller-ctrl-plane-client-identity-secret   49s
openziti-dev-controller-ctrl-plane-identity            False   openziti-dev-controller-ctrl-plane-identity-secret          49s
openziti-dev-controller-ctrl-plane-intermediate-cert   True    openziti-dev-controller-ctrl-plane-intermediate-secret      49s
openziti-dev-controller-edge-signer-cert               True    openziti-dev-controller-edge-signer-secret                  49s
openziti-dev-controller-web-client-identity            False   openziti-dev-controller-web-client-identity-secret          49s
openziti-dev-controller-web-identity-cert              False   openziti-dev-controller-web-identity-secret                 49s
openziti-dev-controller-web-intermediate-cert          True    openziti-dev-controller-web-intermediate-secret             49s
trust-manager                                          True    trust-manager-tls                                           49s

I wont paste the entire get certificaterequest output, but after a few minutes we have 50+ different CSRs with new ones being created every couple minutes or so. All my CSRs using my top level private CA have approved true && ready true

NAME                                                         APPROVED   DENIED   READY   ISSUER                                                   REQUESTOR                                         AGE
openziti-dev-controller-ctrl-plane-client-identity-qkjzr     True                False   openziti-dev-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   3m8s
openziti-dev-controller-ctrl-plane-identity-6tnbd            True                False   openziti-dev-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   3m2s
openziti-dev-controller-ctrl-plane-intermediate-cert-24d9p   True                True    <redacted>                                         system:serviceaccount:cert-manager:cert-manager   2m49s
openziti-dev-controller-ctrl-plane-intermediate-cert-2zfm2   True                True    <redacted>                                         system:serviceaccount:cert-manager:cert-manager   32s
openziti-dev-controller-web-client-identity-zd65w            True                False   openziti-dev-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   3m7s
openziti-dev-controller-web-identity-cert-gj5sn              True                False   openziti-dev-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   3m3s
openziti-dev-controller-web-intermediate-cert-2967t          True                True    <redacted>                                         system:serviceaccount:cert-manager:cert-manager   68s
openziti-dev-controller-web-intermediate-cert-44jrk          True                True    <redacted>                                         system:serviceaccount:cert-manager:cert-manager   84s
openziti-dev-controller-edge-signer-cert-2xc2v               True                True    <redacted>                                         system:serviceaccount:cert-manager:cert-manager   35s
openziti-dev-controller-edge-signer-cert-48tgv               True                True    <redacted>                                         system:serviceaccount:cert-manager:cert-manager   64s
trust-manager-gmdxt                                          True                True    trust-manager                                            system:serviceaccount:cert-manager:cert-manager   3m6s

Perhaps it could be an issue with the PKI tool I am using (hashicorp vault) and the role policies attached to the issuer, as I had to make a role with specific key and name permissions. Maybe I need to check the config if it is permitted to issue intermediate certificates.

The distribution of failures is concentrated on the leaves requested from Ziti's intermediates, whereas Ziti's intermediates that were requested from your alternative issuer succeeded. I should be able to reproduce this if it's not limited to Vault-managed issuers.

For reference, here's a healthy list of CSRs in the ziti namespace following a successful controller deployment with default PKI, not using the alternative issuer.

❯ k -n miniziti get certificaterequests.cert-manager.io
NAME                                             APPROVED   DENIED   READY   ISSUER                                           REQUESTER                                         AGE
ziti-controller-ctrl-plane-client-identity-1     True                True    ziti-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-ctrl-plane-identity-1            True                True    ziti-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-ctrl-plane-intermediate-cert-1   True                True    ziti-controller-ctrl-plane-root-issuer           system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-ctrl-plane-root-cert-1           True                True    ziti-controller-selfsigned-ca-issuer             system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-edge-root-cert-1                 True                True    ziti-controller-selfsigned-ca-issuer             system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-edge-signer-cert-1               True                True    ziti-controller-edge-root-issuer                 system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-web-client-identity-1            True                True    ziti-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-web-identity-cert-1              True                True    ziti-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-web-intermediate-cert-1          True                True    ziti-controller-web-root-issuer                  system:serviceaccount:cert-manager:cert-manager   22h
ziti-controller-web-root-cert-1                  True                True    ziti-controller-selfsigned-ca-issuer             system:serviceaccount:cert-manager:cert-manager   22h

I've started working to reproduce it and haven't yet seen the flurry of CSRs, but I do not see 100% expected behavior with alternative issuers. I tried setting edge and web to use ctrlPlane root CA, and still the web cert is issued by the old web root. I will continue exploring later today or tomorrow. There may be an issue with CSRs or trust bundle or both.

Good morning!

Thanks for looking into this! Today I'll test with the default PKI (no alt issuer) however It would be nice to not have to add another CA to a root store on endpoints.

Just another bit of info for you, the behavior was the same regardless of if I set trust-manager && cert-manager to "enabled: true" or "enabled: false".

The CM/TM "enabled" input values control whether or not those Helm subcharts (dependencies) are also deployed when you deploy the ziti-controller umbrella (parent) chart.

Working with a single PKI, the ctrlPlane root CA, I still didn't encounter a runaway flurry of CSRs, but I noticed that setting alternative issuers can create a deadlock, and addressed that specific problem in Release ziti-controller-1.3.0 · openziti/helm-charts · GitHub

Let's diagnose precisely what's going wrong by referencing this overview of the Cert Manager workflow. A crucial error event is probably on one of CM's custom resources.

In this case, we're only dealing with a custom root, not a public root like LetsEncrypt, so there will be no Orders, Challenges custom resources in play. I think the workflow is like this:

  1. Create (Cluster)Issuer
  2. create Certificate referencing the (Cluster)Issuer
  3. cert Controller creates CertificateRequest, creates an opaque Secret w/ private key in data key tls.key, waits for issuance to update Secret with tls.crt, ca.crt (check cert-manager pod logs for clues to this async process)

Does that match your understanding?

Does tracing the failure this way reveal any new clues?

There's quite a bit of logs, mainly it looks like the custom issuer that the templates create are not found, I confirmed this by running

Here are some examples of those logs

cert-manager/issuers "msg"="issuer in work queue no longer exists" "error"="issuer.cert-manager.io \"openziti-dev-controller-edge-signer-issuer\" not found"
cert-manager/certificaterequests-issuer-venafi "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-web-identity-cert-vwt55\": the object has been modified; please apply your changes to the latest version and try again" "key"="openziti/openziti-dev-controller-web-identity-cert-vwt55"
cert-manager/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-web-client-identity-bk2w8\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/openziti/openziti-dev-controller-web-client-identity-bk2w8, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 58357f48-4360-40ca-9337-18c30be4383a, UID in object meta: " "key"="openziti/openziti-dev-controller-web-client-identity-bk2w8"
cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-web-identity-cert-vwt55\": the object has been modified; please apply your changes to the latest version and try again" "key"="openziti/openziti-dev-controller-web-identity-cert-vwt55"
cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"openziti-dev-controller-web-intermediate-cert\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificates/openziti/openziti-dev-controller-web-intermediate-cert, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 391ecadb-1a68-45ef-aa7a-164c3128b8d9, UID in object meta: " "key"="openziti/openziti-dev-controller-web-intermediate-cert"
cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"openziti-dev-controller-edge-signer-cert\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificates/openziti/openziti-dev-controller-edge-signer-cert, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 43566531-f724-4e45-8c06-ffb33bc37095, UID in object meta: " "key"="openziti/openziti-dev-controller-edge-signer-cert"
cert-manager/certificaterequests-issuer-venafi "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"openziti-dev-controller-ctrl-plane-client-identity-dzzvv\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/openziti/openziti-dev-controller-ctrl-plane-client-identity-dzzvv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 54b86ce6-85cc-470f-873f-fe207d275687, UID in object meta: " "key"="openziti/openziti-dev-controller-ctrl-plane-client-identity-dzzvv"

When running the helm template without alt issuers, all my CSRs are set to Ready True and there is no looping

NAME                                                         APPROVED   DENIED   READY   ISSUER                                                   REQUESTOR                                         AGE
openziti-dev-controller-ctrl-plane-client-identity-rrbc7     True                True    openziti-dev-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   30s
openziti-dev-controller-ctrl-plane-identity-btjbb            True                True    openziti-dev-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   28s
openziti-dev-controller-ctrl-plane-intermediate-cert-6mrkt   True                True    openziti-dev-controller-ctrl-plane-root-issuer           system:serviceaccount:cert-manager:cert-manager   30s
openziti-dev-controller-ctrl-plane-root-cert-f6l95           True                True    openziti-dev-controller-selfsigned-ca-issuer             system:serviceaccount:cert-manager:cert-manager   32s
openziti-dev-controller-edge-root-cert-cpbsf                 True                True    openziti-dev-controller-selfsigned-ca-issuer             system:serviceaccount:cert-manager:cert-manager   33s
openziti-dev-controller-edge-signer-cert-j278c               True                True    openziti-dev-controller-edge-root-issuer                 system:serviceaccount:cert-manager:cert-manager   31s
openziti-dev-controller-web-client-identity-ls6pk            True                True    openziti-dev-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   32s
openziti-dev-controller-web-identity-cert-m6mtr              True                True    openziti-dev-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   27s
openziti-dev-controller-web-intermediate-cert-k25lv          True                True    openziti-dev-controller-web-root-issuer                  system:serviceaccount:cert-manager:cert-manager   32s
openziti-dev-controller-web-root-cert-4qq5d                  True                True    openziti-dev-controller-selfsigned-ca-issuer             system:serviceaccount:cert-manager:cert-manager   32s

however now I am running into another issue as the openziti controller pod is stuck in init state

 Warning  FailedMount             4m35s (x5 over 4m43s)  kubelet                  MountVolume.SetUp failed for volume "ziti-controller-ctrl-plane-cas" : configmap "openziti-dev-controller-ctrl-plane-cas" not found
  Warning  FailedMount             4m35s (x5 over 4m43s)  kubelet                  MountVolume.SetUp failed for volume "cert-ctrl-plane-identity" : secret "openziti-dev-controller-ctrl-plane-identity-secret" not found
  Warning  FailedMount             4m35s (x5 over 4m43s)  kubelet                  MountVolume.SetUp failed for volume "cert-web-client-identity" : secret "openziti-dev-controller-web-client-identity-secret" not found
  Warning  FailedMount             4m35s (x5 over 4m43s)  kubelet                  MountVolume.SetUp failed for volume "cert-ctrl-plane-client-identity" : secret "openziti-dev-controller-ctrl-plane-client-identity-secret" not found
  Warning  FailedMount             4m35s (x5 over 4m43s)  kubelet                  MountVolume.SetUp failed for volume "cert-web-identity" : secret "openziti-dev-controller-web-identity-secret" not found

I confirmed the secrets its trying to mount exist

openziti-dev-controller-admin-secret                        Opaque               2      2m39s
openziti-dev-controller-ctrl-plane-client-identity-secret   kubernetes.io/tls    3      92s
openziti-dev-controller-ctrl-plane-identity-secret          kubernetes.io/tls    3      92s
openziti-dev-controller-ctrl-plane-intermediate-secret      kubernetes.io/tls    3      2m13s
openziti-dev-controller-ctrl-plane-root-secret              kubernetes.io/tls    3      2m32s
openziti-dev-controller-edge-root-secret                    kubernetes.io/tls    3      2m33s
openziti-dev-controller-edge-signer-secret                  kubernetes.io/tls    3      2m30s
openziti-dev-controller-trust-domain                        Opaque               1      2m39s
openziti-dev-controller-web-client-identity-secret          kubernetes.io/tls    3      92s
openziti-dev-controller-web-identity-secret                 kubernetes.io/tls    3      92s
openziti-dev-controller-web-intermediate-secret             kubernetes.io/tls    3      2m13s
openziti-dev-controller-web-root-secret                     kubernetes.io/tls    3      2m32s

I'll try with the new release of 1.3.0 and report back!

1 Like

I'm still experiencing the issue with your controller chart 1.3.0.

I'm sure I have a config issue somewhere, as you couldn't reproduce the issue.

Thanks again for your help and I'll report back if I find a solution.

I wasn't able to reproduce with an alternative ClusterIssuer. Here's what my inputs looked like for the successful test.

I started with a fresh instance of K8s v1.35 w/ latest CM/TM.

I created a ClusterIssuer named "new-root-clusterissuer" to serve as my single root CA for Ziti's three intermediates.

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: new-root-selfsigned
spec:
  selfSigned: {}

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: new-root-clusterissuer
spec:
  ca:
    secretName: new-root-cert-secret

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: new-root-cert
  namespace: cert-manager
spec:
  commonName: new-root-cert
  secretName: new-root-cert-secret
  isCA: true
  duration: 87840h
  renewBefore: 720h
  usages:
    - digital signature
    - cert sign
    - crl sign
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    kind: ClusterIssuer
    name: new-root-selfsigned

I deployed the Ziti controller with subcharts disabled since I already installed CM/TM. I used input values like these to base Ziti's intermediates on my alternative root, the same root for all three in this case.

ctrlPlane:
  alternativeIssuer:
    kind: ClusterIssuer
    name: new-root-clusterissuer
edgeSignerPki:
  alternativeIssuer:
    kind: ClusterIssuer
    name: new-root-clusterissuer
webBindingPki:
  alternativeIssuer:
    kind: ClusterIssuer
    name: new-root-clusterissuer
❯ k get certificaterequests.cert-manager.io
NAME                                             APPROVED   DENIED   READY   ISSUER                                           REQUESTER                                         AGE
ziti-controller-ctrl-plane-client-identity-1     True                True    ziti-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   171m
ziti-controller-ctrl-plane-identity-1            True                True    ziti-controller-ctrl-plane-intermediate-issuer   system:serviceaccount:cert-manager:cert-manager   171m
ziti-controller-ctrl-plane-intermediate-cert-1   True                True    new-root-clusterissuer                           system:serviceaccount:cert-manager:cert-manager   171m
ziti-controller-edge-signer-cert-1               True                True    new-root-clusterissuer                           system:serviceaccount:cert-manager:cert-manager   171m
ziti-controller-web-client-identity-1            True                True    ziti-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   171m
ziti-controller-web-identity-cert-1              True                True    ziti-controller-web-intermediate-issuer          system:serviceaccount:cert-manager:cert-manager   171m
ziti-controller-web-intermediate-cert-1          True                True    new-root-clusterissuer                           system:serviceaccount:cert-manager:cert-manager   171m

Coincidentally, Vault was mentioned again to me and now I'm curious about their integration with Cert Manager.

Are you using CM's Issuer spec like this to delegate issuance to Vault?

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: one-root-to-rule-them-all
spec:
  vault:
    path: pki_int/sign/example-dot-com
    server: https://vault.local
    caBundle: <base64 encoded CA Bundle PEM file>
    auth:
      ...

ref: Vault - cert-manager Documentation

Hey!

Yea my cluster issuer looks like this

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: <Issuer-Name>
spec:
  vault:
    auth:
      appRole:
        path: approle
        roleId: <redacted>
        secretRef:
          key: <redacted>
          name: <redacted>
    caBundle: <base64 bundle pem file>
    path: pki_int/sign/example-dot-com
    server: https://vault-server.domain
1 Like

Thank you for confirming that. I will eventually try to reproduce this with a Hashicorp Vault delegated issuer. It'd be good to know if we need to do something in particular to support their integration with Cert Manager. I was not able to reproduce the issue that you encountered by using another non-Vault issuer.

Thank you for looking into it! I did some experimentation last week without much luck. I think the good news is the issue lies in Vault so if I find any solution I will report back.

I forgot to mention what I had stumbled upon last week.

cert-manager/issuers/setup "msg"="signing certificate is not a CA" "error"=null "related_resource_kind"="Secret" "related_resource_name"="openziti-dev-controller-web-intermediate-secret" "related_resource_namespace"="openziti" "resource_kind"="Issuer" "resource_name"="openziti-dev-controller-web-intermediate-issuer" "resource_namespace"="openziti" "resource_version"="v1"

Describe on that issuer

Spec:
  Ca:
    Secret Name:  openziti-dev-controller-web-intermediate-secret
Status:
  Conditions:
    Message:               Error getting keypair for CA issuer: certificate is not a CA

Above is one of the errors I was getting, which leads me to figure out why vault is not issuing a CA cert to the custom issuers listed out in the values file. Most likely vault is configured not to issue out CA certs to cert-manager, but I need to figure out why. I made a brand new clusterissuer from scratch with no restrictions to test and I got the same issue.

(I took a peek at the secret/certificate it issued, and it was indeed missing the CA flag which is odd because the INT ca for that issuer does have subject type CA)

I think the cluster-dot-com role I am using is missing something personally.

1 Like

That's a promising lead!

1 Like