Attempting to use ziti-controller Helm chart: `panic: could not generate default trust domain`

Hello all, I am getting an odd error that I'm unsure how to address when attempting to use the Helm chart ziti-controller for the first time.

panic: could not generate default trust domain: error generating default trust domain from root CA: no root CA detected after chain assembly from the root identity server cert and ca bundle
goroutine 1 [running]:
github.com/openziti/ziti/controller/config.LoadConfig({0x7fff108771cb?, 0xc0008078c0?})
	github.com/openziti/ziti/controller/config/config.go:383 +0x3445
github.com/openziti/ziti/ziti/controller.run(0xc000c67100?, {0xc000c12e50, 0x1, 0x3ad5116?})
	github.com/openziti/ziti/ziti/controller/run.go:54 +0x54a
github.com/spf13/cobra.(*Command).execute(0xc000aff508, {0xc000c12e20, 0x1, 0x1})
	github.com/spf13/cobra@v1.8.1/command.go:989 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0x5b33880)
	github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/openziti/ziti/ziti/cmd.Execute()
	github.com/openziti/ziti/ziti/cmd/cmd.go:81 +0x1a
main.main()
	github.com/openziti/ziti/ziti/main.go:51 +0xf
my values.yaml
clientApi:
  advertisedHost: ziti.mydnsname.com
  advertisedPort: 443
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: gce
  dnsNames: []
cert-manager:
  enabled: true
  enableCertificateOwnerRef: true
  installCRDs: true
trust-manager:
  enabled: true
  app:
    trust:
      namespace: mynamespace
  crds:
    enabled: true

I'm using the full values.yaml found in the Helm repository, but left everything default except for what's provided above. I already have cert-manager installed in this cluster; I've tried disabling it and get the same result. Though it shouldn't theoretically matter, I'm using ArgoCD to deploy this. All the other resources seem to have created successfully without issue. Here is the logs from the chart created cert-manager pods:

cert-manager logs

cert-manager-cainjector:

I0803 23:51:56.928667       1 start.go:126] "starting" version="v1.11.5" revision="b1d501c85d97afd2adde2e86c2b119ac060caece"
I0803 23:51:57.979767       1 request.go:690] Waited for 1.017148344s due to client-side throttling, not priority and fairness, request: GET:https://10.5.0.1:443/apis/external-secrets.io/v1alpha1?timeout=32s
I0803 23:51:59.029855       1 request.go:690] Waited for 2.065666932s due to client-side throttling, not priority and fairness, request: GET:https://10.5.0.1:443/apis/servicedirectory.cnrm.cloud.google.com/v1beta1?timeout=32s
I0803 23:52:00.030036       1 request.go:690] Waited for 3.065287848s due to client-side throttling, not priority and fairness, request: GET:https://10.5.0.1:443/apis/dlp.cnrm.cloud.google.com/v1beta1?timeout=32s
I0803 23:52:01.079786       1 request.go:690] Waited for 4.114448216s due to client-side throttling, not priority and fairness, request: GET:https://10.5.0.1:443/apis/storagetransfer.cnrm.cloud.google.com/v1beta1?timeout=32s
I0803 23:52:01.287360       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...

cert-manager:

I0803 23:51:57.135245       1 start.go:75] cert-manager "msg"="starting controller" "git-commit"="b1d501c85d97afd2adde2e86c2b119ac060caece" "version"="v1.11.5"
I0803 23:51:57.135691       1 controller.go:242] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.5.0.10:53"]
W0803 23:51:57.135791       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0803 23:51:57.137077       1 controller.go:70] cert-manager/controller "msg"="enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]" 
I0803 23:51:57.137538       1 controller.go:134] cert-manager/controller "msg"="starting leader election" 
I0803 23:51:57.137764       1 controller.go:91] cert-manager/controller "msg"="starting metrics server" "address"={"IP":"::","Port":9402,"Zone":""}
I0803 23:51:57.138182       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-controller...

cert-manager-webhook:

I0803 23:51:58.919652       1 feature_gate.go:249] feature gates: &{map]}
W0803 23:51:58.919733       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0803 23:51:58.932750       1 webhook.go:129] cert-manager "msg"="using dynamic certificate generating using CA stored in Secret resource" "secret_name"="net-edge-prod-cert-manager-webhook-ca" "secret_namespace"="net-edge-prod"
I0803 23:51:58.933375       1 server.go:133] cert-manager/webhook "msg"="listening for insecure healthz connections" "address"=":6080"
I0803 23:51:58.933468       1 server.go:197] cert-manager/webhook "msg"="listening for secure connections" "address"=":10250"
I0803 23:51:59.940339       1 dynamic_source.go:266] cert-manager/webhook "msg"="Updated cert-manager webhook TLS certificate" "DNSNames"=["net-edge-prod-cert-manager-webhook","net-edge-prod-cert-manager-webhook.net-edge-prod","net-edge-prod-cert-manager-webhook.net-edge-prod.svc"]

I would be extremely grateful for any thoughts or troubleshooting steps anyone could provide.

I'll try to reproduce this with the latest charts. I'm unsure what the issue is at first glance. Will you try the following in hopes of narrowing the cause?

1. Ensure you have updated the Helm repo and installed latest charts.
2. Try installing the CRDs separately in advance, then the controller chart with CM and TM installCRDs: false and crds.enabled: false, respectively.

EDIT: I reproduced this with latest chart and it was also presenting in a CI test which I failed to notice, unfortunately. :disappointed: I have just changed the default branch protection rule to require "deploy to minikube."

1 Like

The earliest incompatible binary was 1.1.6. I'll release a controller chart version as soon as I understand how the controller's config YAML needs to be updated for the new, required "trust domain." (link to changelog).

The workaround is to override the controller's image like this in Helm values.

image:
    tag: 1.1.5
2 Likes

Great find! Thanks for this. I was able to get around that error by overriding the version.

Getting a different error now but I think this one is user error :laughing:

fix for review here: generate a spiffe id if trustDomain not specified by qrkourier · Pull Request #246 · openziti/helm-charts · GitHub

1 Like

Controller chart release:

1 Like