Lets Encrypt Certs with Helm on AltIngress Issues

Support
1

Hi Openziti crew,

I hope to find you well!

I have been working on my Helm deployment for a while now. I've gotten it working where I can tunnel to services running on my k8s cluster using my private altIssuers (with your help from this ticket OpenZiti Helm Chart - Using Enterprise PKI Instead of Self-Signed CA - #21 by Tetrusp)

I am onto a new issue, and I am curious what you guys think / if there is a solution.

In short, I would like to use Lets Encrypt certs on AltIngress objects so I don't expose my private PKI to the internet. Is that possible?

I originally wanted to try and see if I can have openziti sit behind a traditional VPN of sorts so that I wouldn't need to worry about that but I haven't been able to get that to work due to client side routing probably getting messed up with 2 Tunnels running on the same Windows machine trying to send the user to the same endpoint. There is probably some routing magic I could try but for now I am deciding i'll test with Openziti having a public controller and router.

Here comes my issue, I cant seem to get my Lets Encrypt certs to attach to the AltIngress hosts specified in my helm charts.

I know my Certmanager cluster-issuer using LE works as I see the certificate set as Ready. I also confirmed the tls.crt && tls.key are mounted on the controller pod, (i popped a shell to confirm). But when I navigate to my altingress hostname, I see my private PKI cert, not the LE cert.

I've tried many different iterations of my helm chart without success (disabling altingress, disabling altissuer, manually setting different hostnames, etc, etc,). I am assuming I am doing something wrong so here are my helm bits.

(I've tried playing around with the tls portion where there is a comment about AltServerCerts but anytime I put stuff in there the controller pod would not deploy, also enabling alt ingress on the mgt api would cause the controller pod to not deploy either)

clientApi:
  containerPort: 1280
  advertisedHost: "ziti-controller.<redacted>"
  advertisedPort: 443
  service:
    enabled: true
    type: ClusterIP 
  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations: 
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  altIngress:
    enabled: true
    advertisedHost: "ziti-controller-alt.<redacted>"
    ingressClassName: "nginx"
    annotations: 
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    # certificate to project into the container in webBindingPki.altServerCerts
       tls: {}
  dnsNames: ["ziti-controller-alt.<redacted>"]

managementApi:
  containerPort: 1281
  advertisedHost: "ziti-controller-mgt.<redacted>"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP  
  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations: 
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  altIngress:
    enabled: false

ctrlPlane:
  service:
    enabled: true
    type: ClusterIP
  ingress:
    enabled: false
  alternativeIssuer: 
    kind: ClusterIssuer
    name: <redacted>

edgeSignerPki:
  enabled: true
  alternativeIssuer: 
    kind: ClusterIssuer
    name: <redacted>

webBindingPki:
  enabled: true
  altServerCerts: 
    - mode: certManager
      secretName: ziti-controller-alt-server-cert
      dnsNames:
        - "{{ .Values.clientApi.altIngress.advertisedHost }}"
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: <Lets Encrypt Issuer>
      mountPath: /etc/ziti/alt-server-cert
  alternativeIssuer: 
    kind: ClusterIssuer
    name: <redacted>

Here is my certs output

NAME                                                   READY   SECRET                                                      AGE
openziti-dev-controller-alt-cert-0                     True    ziti-controller-alt-server-cert                             3m34s
openziti-dev-controller-ctrl-plane-client-identity     True    openziti-dev-controller-ctrl-plane-client-identity-secret   3m34s
openziti-dev-controller-ctrl-plane-identity            True    openziti-dev-controller-ctrl-plane-identity-secret          3m34s
openziti-dev-controller-ctrl-plane-intermediate-cert   True    openziti-dev-controller-ctrl-plane-intermediate-secret      3m34s
openziti-dev-controller-edge-signer-cert               True    openziti-dev-controller-edge-signer-secret                  3m33s
openziti-dev-controller-web-client-identity            True    openziti-dev-controller-web-client-identity-secret          3m33s
openziti-dev-controller-web-identity-cert              True    openziti-dev-controller-web-identity-secret                 3m33s
openziti-dev-controller-web-intermediate-cert          True    openziti-dev-controller-web-intermediate-secret             3m33s
trust-manager                                          True    trust-manager-tls                                           3m33s
2

So I am dummy, the SAN in the clientApi was causing the issue, since i guess the Alt ingress fqdn was being added to the privatePKI, the app wasn't using the alt_server_cert.

Cheers!

1 Like
3

Circling back on this,

While the alt_server_certs works, it seems like it doesn't necessarily solve my problem of trying to not expose private PKI on the internet, as the controller still publishes the non alt FQDN for a desktop edge client to communicate with under alt-controller.domain/network-jwts (in the jwt, the claim points to the non alt domain). And the Windows Client gives me a timeout as I assume its trying to reach the non alt interface even if I specify the Alt in the controller URL.

So, is there any way to have the ziti controller / router not expose private PKI when connecting to the overlay?

Having a ziti network be fully private is one way but I guess that would need a gateway/proxy of some sort, I was hoping to be able to have the router/controller use lets encrypt certs for their public facing interfaces, and once a client connects the service switches over to the private PKI.

Let me know what you guys think, and thanks for your time :slight_smile: