Hi , I am using a domain to host the ziti controller with an ssl using cert-manager and letsencrypt .
I see the certifcate been created succesfully .
But when I visit the zac console I get warnings in browser that Your connection is not private.
Is that intended for ssl-passthrough ?
when I use curl -v I also get local issuer not found . But when I edit ssl-passthrough: false I no longer face that
OpenZiti always runs it's own pki, established during the very first installation steps. The controller will always listen on a port for this FQDN.
When you want to use LetsEncrypt, these will ALWAYS be what we call "alternate server certs". They MUST be from a different FQDN than the self signed pki. If they are not, and if they are identical, you can get the behavior your describing.
My expectations are either you used the exact same domain name, or you are not successfully passing through TLS.
You can verify this using openssl s_client and inspecting the returned chain.
Here the returned output
ingress git:(main) ✗ openssl s_client -connect zitictrl3.stratscient.com:443 -showcerts
Connecting to 34.72.225.57
CONNECTED(00000005)
depth=1 CN=ziti-controller-web-intermediate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=ziti-controller-web-identity
verify return:1
---
Certificate chain
0 s:CN=ziti-controller-web-identity
i:CN=ziti-controller-web-intermediate
a:PKEY: rsaEncryption, 4096 (bit); sigalg: ecdsa-with-SHA256
v:NotBefore: Jan 11 19:17:22 2025 GMT; NotAfter: Jan 19 19:17:22 2035 GMT
-----BEGIN CERTIFICATE-----
xx
-----END CERTIFICATE-----
---
Server certificate
subject=CN=ziti-controller-web-identity
issuer=CN=ziti-controller-web-intermediate
---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2420 bytes and written 430 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 4096 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_128_GCM_SHA256
Session-ID: 0596AC2665763907CC7F49A48D14DF8534D266BC4F8313AF8491B59B8ABF7369
Session-ID-ctx:
Resumption PSK: 655D3584167B5DCE7AB3FBB5EA22BED6574B54CC8BA2E4CA5E68020295B1BAB0
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - 74 38 63 cd d8 f5 63 bb-79 ad 55 3b 99 22 3d 4b t8c...c.y.U;."=K
0010 - 72 b4 24 ad 98 54 09 1a-f9 48 6f ba d6 73 86 58 r.$..T...Ho..s.X
0020 - 7e 24 a2 c4 b3 8e a4 6f-85 b4 4e 48 3e 56 37 44 ~$.....o..NH>V7D
0030 - 3e 09 2e 2b 48 25 01 c4-3f b0 1a 00 18 b6 0d 44 >..+H%..?......D
0040 - 35 ac ea 39 ac 30 1b 79-69 1a e4 0b 59 1d 07 e0 5..9.0.yi...Y...
0050 - ce ae e8 e9 f0 be 1f 26-17 b7 9b 6f 8e 6a c1 f0 .......&...o.j..
0060 - bd 0e c5 3a 77 f2 5a ad-61 ...:w.Z.a
Start Time: 1736687787
Timeout : 7200 (sec)
Verify return code: 20 (unable to get local issuer certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
closed
➜ ingress git:(main) ✗
Clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: ziti
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: my_email
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
- dns01:
cloudflare:
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
selector:
dnsNames:
- "*.stratscient.com"
The ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.allow-http: "false"
meta.helm.sh/release-name: ziti-controller
meta.helm.sh/release-namespace: ziti
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
creationTimestamp: "2025-01-11T19:17:02Z"
generation: 3
labels:
app.kubernetes.io/instance: ziti-controller
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ziti-controller
app.kubernetes.io/version: 1.1.15
helm.sh/chart: ziti-controller-1.1.5
name: ziti-controller-ctrl
namespace: ziti
resourceVersion: "798529"
uid: 605120f7-8359-4be5-97d0-8fc735294aff
spec:
ingressClassName: nginx
rules:
- host: zitictrl3.stratscient.com
http:
paths:
- backend:
service:
name: ziti-controller-ctrl
port:
number: 443
path: /
pathType: Prefix
tls:
- hosts:
- zitictrl3.stratscient.com
secretName: tls-cert1
status:
loadBalancer:
ingress:
- ip: 34.72.225.57
I thought ziti controller needs to allow a way to pass the certs so It can handel the ssl-passthrough , on my external domain hosting the ziti
The certificate returned from the zac is overlapping the wildcard DNS specified in Clusterissuer.yaml. you're using the same FQDN for both.
I would recommend you create the network using a longer FQDN to avoid overlapping the wildcard, or make the wildcard longer so it won't overlap.
For OpenZiti to work, it must terminate TLS at the controller. That's usually done via the ingress/proxy/loadbalancer (aka kubernetes or the like). It's not OpenZiti itself that passes TLS though.
But the certificates are created different for each domain , although I tried full FQDN at starting and had same results . Since removing ssl-passthrough makes it work I was not bothering the issues around certs and ingress .
Ya but k8s deployment for controller needs ssl-passthrough which means the loadbalancer would not terminate TLS and needs the backend service to terminate instead , that was my understand though
The symptom is the console is presenting the wrong certificate, correct?
Is zitictrl3.stratscient.com the domain name you wish to use for the console?
Is zitictrl3.stratscient.com also configured for any other ziti-controller web binding? It must be unique.
The Ingress YAML you shared has contradicting properties. It asks the Cert Manager to issue and bind a certificate to terminate TLS with NGINX and has the mutually exclusive passthrough annotation. As you correctly mentioned, an Ingress must pass through TLS to the ziti-controller pod, not terminate at the cluster edge, e.g., with an Ingress Controller.
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
# ...
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
# ...
tls:
- hosts:
- zitictrl3.stratscient.com
secretName: tls-cert1
Since you have a Cert Manager issuer configured, you can change your ziti-controller release's input values to request a trusted certificate from that issuer for a unique domain name. You must not use the same domain name for the console's trusted certificate and the controller's client API or management API (i.e., the other web bindings).
The following example from the ziti-controller chart's README demonstrates the input values that automatically create the alternative Ingress resource, which you may use to access the console, and also to request a certificate for that Ingress from your Issuer. This example uses console.ziti.example.com as the alternative domain name for the console, and edge.ziti.example.com for the ziti-controller's primary web bindings, i.e., the client and mgmt APIs, which are combined on a single binding in this example.
clientApi:
advertisedHost: edge.ziti.example.com
ingress:
enabled: true
ingressClassName: nginx
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
service:
enabled: true
type: ClusterIP
altIngress:
enabled: true
ingressClassName: nginx
advertisedHost: console.ziti.example.com
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
webBindingPki:
enabled: true
altServerCerts:
- mode: certManager
secretName: my-alt-server-cert
dnsNames:
- console.ziti.example.com
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: cloudflare-dns01-issuer
mountPath: /etc/ziti/alt-server-cert
This is the most automatic way to create both an Ingress and alt server cert, but you could craft your own Ingress or LoadBalancer Service resource instead, as long as the ziti-controller's web PKI has an alternative server cert with a unique DNS SAN.
Thank you @qrkourier .
To be clear . There are 2 domains I use . ziticlient3.stratscient.com for the client url which we often use for login and zitictrl3.stratscient.com .
I did hop over the altingress feature but the goal was to show users the ssl security on the primary ingress .
Let's say I give them the controller and client url to register their routers . If they check the ssl validation via curl -v https://zitictrl3.stratscient.com or curl -v https://ziticlient3.stratscient.com it must show it valid .
You can verify a Ziti TLS server certificate only with the Ziti trust bundle, not an OS or web browser trust bundle.
You can verify a Ziti TLS alt server certificate with an OS or web browser trust bundle if the certificate is issued by a publicly trusted authority like Let's Encrypt.
How does the alt ingress work ? Is it complete replacement for the primary ingress host if I want external clients to verify the certificates ?
Which means I can forget about the main client api url and use the alt ingress url since it gets verified with ssl ?
It's an Ingress resource created by the ziti-controller chart for the domain name of the alt server cert.
No, it's always separate from what you called the primary ingress host. Ziti's primary web bindings (the client API and mgmt API) always have Ziti-internal TLS certificates for use with Ziti identities and routers, never publicly-trusted TLS certificates from a publicly-trusted authority like Let's Encrypt. This is why the pod must terminate TLS.
External clients, if you mean cURL or a web browser using a generic trust store, can verify a Ziti TLS alt server certificate with an OS or web browser trust bundle if the certificate is issued by a publicly trusted authority like Let's Encrypt. A generic trust store can never be used to verify a Ziti TLS server certificate for a primary web binding or router control plane, i.e., the TLS servers utilized by Ziti identities and routers.
No, Ziti identities and routers must use Ziti's internal PKI, not a publicly-trusted PKI like Let's Encrypt.
However, the definition of "internal PKI" is more flexible than it may seem. I simply mean "not a publicly-trusted PKI." That is, Ziti's PKI may be unique to Ziti or it may be derived from another private PKI that you fully control. It must not be shared with an untrusted party.
Ok fair enough , I did learn that through ssl-passthrough the ingress can't do path based routing since it does not decrypt the data received . But I wanted to block the access to ZAC console .
How do we disable it ?
That's true. The load balancer/reverse proxy can't inspect the application layer when passing through TLS to the pod, so it can't enforce HTTP header rules.
The console is an SPA served from the mgmt API, so it has (almost) exactly the same attack surface as the mgmt API itself. Most likely, you want to block access to the mgmt API by setting ziti-controller chart inputs like these, which will bind the mgmt API to a different TCP port for which you can configure an Ingress, firewall rule, ACL, security group, etc. to control access separately from ziti-controller's outward-facing client API web binding.
managementApi:
advertisedHost: mgmt.ziti.example.com
advertisedPort: 443
containerPort: 1281
service:
enabled: true
type: ClusterIP
Link to doc section about securing the mgmt API: Install the Controller in Kubernetes | OpenZiti
Your console and CLI login URLs will change with this configuration, and the console will no longer be available on the same base URL as the client API.
After upgrading the Helm release to split the mgmt API to a separate web binding, you can always check the new console URL like this.
helm get notes ziti-controller
cool , so to confirm once I split them and have mgmnt service. I no longer interact with client api like the way I was doing ( creating identities , routers , services ) I only use management api ?
but still considering our ziti urls would not have firewalls just to us , other users would still access the UI
It would be great , if we can have an option to completely disable UI to us and our users . As we don't use API and rely on ziti cli and apis most of the time
Ziti identities and routers will continue to use the same old client API and router ctrl plane URLs (both are by default: {{ .Values.clientApi.advertisedHost }}:{{ .Values.clientApi.advertisedPort }}).
The Ziti CLI uses the mgmt API exclusively to manage Ziti edge entities, etc. The console is always bound to the mgmt API. This means that splitting the mgmt API to a different port requires a new mgmt API URL for CLI and console.
Here's a more specific example.
managementApi:
advertisedHost: mgmt.ziti.example.com
advertisedPort: 443
containerPort: 1281
ingress:
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
enabled: true
ingressClassName: nginx
service:
enabled: true
type: ClusterIP
With these inputs, your CLI login command changes like this:
ziti edge login mgmt.ziti.example.com:443
...and your console URL changes to https://mgmt.ziti.example.com/zac/.
This means you control access for CLI and console separate from the access used by Ziti identities (users, devices, etc.) and Ziti routers, and so the console is unavailable unless you have separately authorized access to the new mgmt URL.
Still, adding an input value to disable the console binding on the mgmt API is easy. This does not significantly reduce the attack surface.
Ok seems like by splitting mgmt service the console component from client is taken away by mgmt service .. And I can control the access to mgmt in my domain to control the access to console .
And the client api which we use need to have permission on mgmt in order to create any identities and services since it talks to mgmt .
That way it seems to solve the problem of hiding the consol and also much secure way to deal with admin tasks .
I did try the alt ingress for clientapi . Wondering if there is one for ctrl plane as well