Was the Ingress resource created with the expected values for the mgmt API you attempted to publish with Traefik?
If the Traefik Ingress for clientApi is functioning, then I assume you will need the same types of annotations for managementApi's Ingress.
Ensure the managementApi.advertisedHost is a domain name that resolves to Traefik's external IP.
All looks correct to me:
kubectl -n ziti get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ziti-controller-mgmt traefik zac.domain.com 80, 443 18h
dig zac.domain.com
;; ANSWER SECTION:
zac.domain.com. 3600 IN A internal.ip.address.here
I think the issue is that the console seems to be refusing connections when it runs as a separate service.
The fact that doing a port-forward does not allow connections is confusing.
Something is still misconfigured. Are you running the latest chart version?
This works:
kubectl -n miniziti port-forward services/ziti-controller-mgmt 8443:443
then
kubectl get secrets "ziti-controller-admin-secret" \
--output go-template='{{index .data "admin-password" | base64decode }}' \
| xargs ziti edge login localhost:8443 \
--yes --username admin \
--password
or https://localhost:8443/zac/ in a browser.
Here's my ziti-controller-mgmt ingress for the inress-nginx controller.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.allow-http: "false"
meta.helm.sh/release-name: ziti-controller
meta.helm.sh/release-namespace: miniziti
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
creationTimestamp: "2024-10-30T15:21:42Z"
generation: 1
labels:
app.kubernetes.io/instance: ziti-controller
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ziti-controller
app.kubernetes.io/version: 1.1.15
helm.sh/chart: ziti-controller-1.1.1
name: ziti-controller-mgmt
namespace: miniziti
resourceVersion: "872293"
uid: 0c40ab5c-ed8d-4b5f-ae06-e7f448a90320
spec:
ingressClassName: nginx
rules:
- host: miniziti-controller-mgmt.192.168.49.2.sslip.io
http:
paths:
- backend:
service:
name: ziti-controller-mgmt
port:
number: 443
path: /
pathType: Prefix
status:
loadBalancer:
ingress:
- ip: 192.168.49.2
Are you able to resolve the domain name you set in .Values.managementApi.advertisedHost, .e.g, miniziti-controller-mgmt.192.168.49.2.sslip.io, to the external IP address reported in the Ingress status, e.g. 192.168.49.2?
I just upgraded the controller chart from version 1.0.16 to 1.1.1.
I did test the port forward again, and it functioned properly this time 
When I try to access it through traefik, the controller logs show this:
{"_context":"tls:0.0.0.0:1281","error":"remote error: tls: bad certificate","file":"github.com/openziti/transport/v2@v2.0.146/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.160:55824","time":"2024-10-30T21:15:14.042Z"}
I did verify the certificate:
kubectl -n ziti get certificates
zac-domain-com-tls True zac-domain-com-tls 17m
I have the following section for tls:
tls:
- hosts:
- zac.domain.com
secretName: zac-domain-com-tls
However, in version 1.1.1 of the controller chart, it now shows this error:
coalesce.go:286: warning: cannot overwrite table with non table for ziti-controller.managementApi.ingress.tls (map[])
Yes, I am able to resolve the host. I pasted the output of a dig command one comment above
Great! You've separated the client and mgmt APIs, and you can only access the mgmt API through a port forward because something's wrong with the ingress.
You have a DNS record like this, where the IP matches the ADDRESS column from get ingresses.
Are you trying to configure a separate server certificate for the console? The console shares a web listener with the mgmt API which has a default certificate from Ziti's web PKI, so it's not strictly necessary to configure TLS to begin using the console.
If you want to configure a trusted certificate, e.g., from LetsEncrypt, see this section of the doc about alternative certs, but let's tackle that after we get it working with the default certificate.
The controller logs this message every time it receives any connection that is not mTLS, which includes all legitimate connections to the client and management APIs that do not involve client certificate authentication, including many ziti CLI and console connections.
With the output of get ingress, there is nothing in the ADDRESS column, but that is the same for every service I currently have running successfully.
It is all done via hostname.
The DNS record for zac.domain.com matches the internal IP address that I have set with a LoadBalancer, assigned to traefik.
I am indeed trying to use cert-manager.
I have a ClusterIssuer configured, and I have this as an annotation:
cert-manager.io/cluster-issuer: name-goes-here
Thank you for the link!
It looks like I need this section: Install the Controller in Kubernetes | OpenZiti
additionalVolumes:
- name: my-alt-server-cert
volumeType: secret
mountPath: /etc/ziti/my-alt-server-cert
secretName: my-alt-server-cert
webBindingPki:
altServerCerts:
- mode: secret
secretName: my-alt-server-cert
As far as the helm/tls/table error, do I need to use special formatting for the tls section?
I haven't forgotten about this! You reminded me that it's been a while since I deployed Ziti in k3s, so I'm working on an example that resembles your case.
Wonderful, thank you!
I am testing out STUNner as a solution for call signalling.
I have a service set up that intercepts stun.domain.com on UDP port 3478.
Calls succeed while connected to Wifi on android, but if I connect to openziti through only mobile data, calls do not connect.
I think I need something that allows 192.168.0.0/16 (for LAN) and 10.0.0.0/16 with the port range 32769-65535 to connect to each other.
After ensuring you understand the networking requirements for 1:1 peer calling, yes! You can configure a Ziti service to intercept a CIDR or domain name pattern for one or a range of ports for TCP, UDP, or both. Here is an example intercept.v1 config.
{
"name": "zone2-intercept",
"configTypeId": "g7cIWbcGg",
"data": {
"portRanges": [
{
"low": 32769,
"high": 65535
}
],
"addresses": [
"*.zone2.example.ziti.internal",
"172.16.0.0/12"
],
"protocols": [
"tcp",
"udp"
]
},
"tags": {}
}
corresponding host.v1 config for the server side's identity:
{
"name": "zone2-host",
"configTypeId": "NH5p4FpGR",
"data": {
"forwardProtocol": true,
"forwardAddress": true,
"forwardPort": true,
"allowedAddresses": [
"172.16.0.0/12",
"*.zone2.example.ziti.internal"
],
"allowedPortRanges": [
{
"low": 32769,
"high": 65535
}
],
"allowedProtocols": [
"tcp",
"udp"
],
"httpChecks": [],
"portChecks": []
},
"tags": {}
}
Notably, this example requires that the IP range and domain name pattern are not fictitious destinations on the server side. The hosting Ziti identity will send traffic to the same target address intercepted on the client side. This effectively "bridges" the 172.16/12 destination subnet to the individual Ziti client identities that are authorized to dial this Ziti service.
Re: k3s example
I finished a k3s example using Ansible.
Traefik v2 can no longer configure Ingress resources with passthrough TLS. Instead, Traefik v2 uses a custom resource, IngressRouteTCP, provided by k3s. My example shows creating these to match the advertised addresses of the client and mgmt APIs, e.g., ctrl1-client.ziti.example.com, ctrl1-mgmt.ziti.example.com.
I've split the client and management APIs on separate web listeners to match your use case. This means the console is only reachable via the ctrl1-mgmt URL, not ctrl1-client, and the ziti CLI, too, must use the mgmt URL.
Additionally, I showed how to configure the controller with an alternative server certificate and one additional IngressRouteTCP that matches the alternative host, e.g., ctrl1-alt.ziti.example.com. The controller chart input values ask Cert Manager to issue this additional certificate from the same issuer as the client/mgmt server certificate. Still, you can point this to your own ClusterIssuer to obtain a certificate to, for example, obtain the alt cert from LetsEncrypt to use with BrowZer.
Throughout the example I used *.sslip.io instead of creating DNS records. These magic wildcard records resolve to the IPv4 address in the record name.
The example playbook's tasks save the kubeconfig from the target host on the Ansible controller while changing the cluster server address to the target host's IP. This helped me test by providing a kubeconfig I could use to interact with the deployed K3s cluster, but it is not necessary for deploying K3s.
Traefik v2, which is included in k3s, also supports Gateway API. This would allow you to avoid using Traefik's custom resources and standardize a k3s deployment on Gateway API's TCPRoute resource. Link to the revision of the playbook that uses Gateway experimental channel instead of Traefik's IngressRouteTCP CRD.
I have been doing a tonne of debugging.
I tried your intercept/host combination, but it would not connect.
I ran nc -l -p 32769 on a desktop.
On android, in termux, I ran nmap, pointing to the LAN IP address, but that did not work.
I was able to connect to the desktop from android by setting up a hostname like fake.domain.com, but I cannot connect via IP address.
By default,nmap relies on ICMP ping during the host discovery phase, and will not probe targets that fail to respond. You must disable host discovery to probe Ziti services' TCP ports because Ziti does not intercept ICMP, e.g., nmap -Pn -p32769 172.16.21.71/32. Alternatively, probe the TCP port directly with netcat (or ncat), e.g., nc -vz 172.16.21.71 32769.
Did the client tunneler report an error at the same time the client attempted to connect to the intercept address:port?
Does policy advisor show the hosting identity is allowed to bind, and the client identity is allowed to dial?
It works!
I could not make this up if I wanted to, but the UI/console added a space in the addresses.
I noticed, manually typed it in, and that sorted everything
If trailing whitespace is allowed in any field in the console, i would consider that a bug. Could you file that/reproduce?
I thought it suspect
I did just verify, and yes, it is a bug.
Would you like me to open a bug report in the ziti repository here?
On ziti-console please: GitHub - openziti/ziti-console
Done
@qrkourier I had a thought to test calls when two users are only connected to Ziti via mobile data on android.
Calls only succeed if one user is connected to Wifi/LAN.
If I try to make a call between two users, and they are connected to Ziti via only mobile data, the connection fails.
I have added 192.0.0.0/8 and 10.0.0.0/8 to the Ziti service.
Is there a different mechanism when connected in this way that would need an additional subnet that I am unaware of?
I think your scenario is two devices both tunneling out to a TURN server that brokers the 1-to-1 audio/video call, and both endpoints are getting configuration for TURN from a Matrix room.
No, I'm not aware of any reason two mobiles couldn't both use 5G, LTE, etc., assuming the bandwidth available supports the application being tunneled.
The most relevant clue will probably be in the log from the tunneler that was attempting to connect with the call failed.
Questions:
192.168.0.0/16 would be safer because 192.0.0.0/8 masks millions of legitimate public IPv4 addresses.