Trust-manager ‘field not declared in schema’ error on fresh EKS + Ziti Controller setup

am3y · January 7, 2025, 8:04pm

I started with a fresh EKS Cluster. Installed ziti-controller using the below commands & the values.yml

Commands

helm repo add openziti https://docs.openziti.io/helm-charts/
helm repo update

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.crds.yaml
kubectl apply -f https://raw.githubusercontent.com/cert-manager/trust-manager/v0.9.0/deploy/crds/trust.cert-manager.io_bundles.yaml

kubectl apply -f storageclass.yml

helm install --namespace ziti-controller --create-namespace ziti-controller-managed openziti/ziti-controller --values controller-values.yml

controller-values.yml


clientApi:
  advertisedHost: ziti-controller.example.com
  service:
    enabled: true
    type: ClusterIP

  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      external-dns.alpha.kubernetes.io/hostname: "ziti-controller.example.com"
      service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # Ensures the LB is public
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-03fef4927363141b8"
      service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"


ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP




highAvailability:
  # -- Ziti controller HA mode
  mode: standalone
  # -- Ziti controller HA swarm replicas
  replicas: 1


persistence:
  enabled: true
  storageClass: "ebs-sc"
  accessMode: ReadWriteOnce
  size: 3Gi


cert-manager:
  enabled: true
  enableCertificateOwnerRef: true
  installCRDs: false

trust-manager:
  enabled: true
  app:
    trust:
      namespace: "ziti-controller"
  crds:
    enabled: false

ingress-nginx:
  enabled: true
  controller:
    extraArgs:
      enable-ssl-passthrough: "true"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-03fef4927363141b8"
        service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"

$ kubectl get pods -n ziti-controller
NAME                                                              READY   STATUS    RESTARTS   AGE
trust-manager-589b9df4d7-wmzj6                                    1/1     Running   0          4m56s
ziti-controller-managed-5fc599bddb-hpwwh                          1/1     Running   0          4m56s
ziti-controller-managed-cert-manager-9d9997797-bz9jf              1/1     Running   0          4m56s
ziti-controller-managed-cert-manager-cainjector-665dc9f48ftp5kl   1/1     Running   0          4m56s
ziti-controller-managed-cert-manager-webhook-5c665ddc4f-4sgt7     1/1     Running   0          4m56s
ziti-controller-managed-ingress-nginx-controller-f7b8784cdgvt2s   1/1     Running   0          4m56s

storageclass.yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

secrets & configmap

$ kubectl get secrets -n ziti-controller
NAME                                                        TYPE                 DATA   AGE
sh.helm.release.v1.ziti-controller-managed.v1               helm.sh/release.v1   1      17m
trust-manager-tls                                           kubernetes.io/tls    3      16m
ziti-controller-managed-admin-client-secret                 kubernetes.io/tls    3      15m
ziti-controller-managed-admin-secret                        Opaque               2      17m
ziti-controller-managed-cert-manager-webhook-ca             Opaque               3      17m
ziti-controller-managed-ctrl-plane-client-identity-secret   kubernetes.io/tls    3      16m
ziti-controller-managed-ctrl-plane-identity-secret          kubernetes.io/tls    3      16m
ziti-controller-managed-ctrl-plane-intermediate-secret      kubernetes.io/tls    3      16m
ziti-controller-managed-ctrl-plane-root-secret              kubernetes.io/tls    3      16m
ziti-controller-managed-edge-root-secret                    kubernetes.io/tls    3      16m
ziti-controller-managed-edge-signer-secret                  kubernetes.io/tls    3      16m
ziti-controller-managed-ingress-nginx-admission             Opaque               3      17m
ziti-controller-managed-trust-domain                        Opaque               1      17m
ziti-controller-managed-web-client-identity-secret          kubernetes.io/tls    3      15m
ziti-controller-managed-web-identity-secret                 kubernetes.io/tls    3      15m
ziti-controller-managed-web-intermediate-secret             kubernetes.io/tls    3      16m
ziti-controller-managed-web-root-secret                     kubernetes.io/tls    3      16m


$ kubectl get configmap -n ziti-controller
NAME                                               DATA   AGE
kube-root-ca.crt                                   1      18m
ziti-controller-managed-config                     4      18m
ziti-controller-managed-ctrl-plane-cas             1      16m
ziti-controller-managed-ingress-nginx-controller   1      18m

When I checked the logs of trust-manager-589b9df4d7-wmzj6 pod, I found one ERROR

E0107 19:45:00.264993       1 controller.go:329] trust/manager "msg"="Reconciler error" "error"="failed to apply bundle status patch: failed to create typed patch object (/ziti-controller-managed-ctrl-plane-cas; trust.cert-manager.io/v1alpha1, Kind=Bundle): .status.target: field not declared in schema" "controller"="bundles" "name"="ziti-controller-managed-ctrl-plane-cas" "namespace"="" "reconcileID"="c797e6ed-e2c4-4ef9-b514-4dad7184b121"

For Reference I have uploaded the whole logfile of that pod.

trust-manager-pod-log.txt (43.9 KB)

Can someone help me to fix this error? Only these things are installed on the EKS Cluster nothing other than that not even routers.

qrkourier · January 8, 2025, 2:08pm

That's a new one. Do you get the same error after downgrading to this version of the Bundle CRD?

https://raw.githubusercontent.com/cert-manager/trust-manager/v0.7.0/deploy/crds/trust.cert-manager.io_bundles.yaml

am3y · January 8, 2025, 2:16pm

Thanks for reply @qrkourier
I tried this.. Still the same

Here are the logs
trust-manager-pod-log2.txt (45.4 KB)

qrkourier · January 8, 2025, 2:25pm

I think we need support from Trust Manager to understand this error reported by Trust Manager's "bundles" controller.go on line 329.

failed to apply bundle status patch: failed to create typed patch object (/ziti-controller-managed-ctrl-plane-cas; trust.cert-manager.io/v1alpha1, Kind=Bundle): .status.target: field not declared in schema

The error indicates TM is failing to patch the Bundle resource with a property that doesn't exist in the schema from the Bundle CRD, .status.target.

Since you're setting up a fresh cluster, will you try deleting the Helm release and TM CRDs, then starting with v0.7.0 CRD before re-creating the Helm release?

am3y · January 8, 2025, 2:26pm

I tried that on another fresh cluster... I'm now installing both cert-manager & trust-manager manually and then testing it. It's just that we need both in the ziti-controller namespace right?

am3y · January 8, 2025, 3:11pm

@qrkourier
So in another fresh EKS Cluster.. I Installed cert-manager & trust-manager using the below commands

helm install cert-manager jetstack/cert-manager \
  --namespace ziti-controller \
  --create-namespace \
  --version v1.16.2 \
  --set crds.enabled=true \
  --set crds.keep=false

helm upgrade trust-manager jetstack/trust-manager \
  --install \
  --version 0.14.0 \
  --namespace ziti-controller \
  --set app.trust.namespace=ziti-controller \
  --wait

Later I installed the ziti-controller using the same above values.yml but I disabled cert-manager & trust-manager subcharts over there..

These are the logs of trust-manager, cert-manager & controller pods..
ziti-controller-log2.txt (5.5 KB)
cert-manager-pod-log.txt (67.1 KB)
trust-manager-pod-log4.txt (12.5 KB)

I guess the issue is resolved, what do you think?

qrkourier · January 8, 2025, 3:42pm

Yes, and thank you for sharing the specific versions of CM and TM you used. There could be a problem with installing Trust Manager CRDs separately when enabling the subchart, and it's good to know that pre-installing Trust Manager for the correct namespace and disabling the subchart is a good solution.

am3y · January 8, 2025, 9:12pm

Can I fix that subchart issue ?

qrkourier · January 8, 2025, 9:17pm

Trust Manager and Cert Manager sub-charts are disabled by default, and it's unclear which conditions introduce the schema deviation you encountered. It's the first report, and I thank you for that. The automated test and my local test haven't triggered that problem, so it doesn't seem to be universal.

am3y · January 8, 2025, 9:18pm

Okay.. I thought I got an first opportunity to contribute to opensource

Topic		Replies	Views
Controller setup on k8s, helm missing configMap Support	3	19	December 31, 2024
Need Guidance on Setting Up Multiple OpenZiti Routers and Identity Devices in EKS	4	37	August 9, 2024
Troubleshooting OpenZiti Tunnel: Edge Router and Service Configuration Issues	27	107	January 7, 2025
Ziti Tunneler Configuration: Resolving Identical ziti0 IPs and Enabling Cross-EC2 Communication Ziti Overlay	4	36	December 28, 2024
How to access private eks cluster apiserver via openziti Ziti Overlay	2	31	July 7, 2024

Trust-manager ‘field not declared in schema’ error on fresh EKS + Ziti Controller setup

Related topics