You switched to mobile data and restarted Ziti before initiating the call, correct?
From the successful call log I see stunner is listening on [::]:3478. Assuming Linux has default dual-stack config, it's listening on all IPv6 and IPv4 interface addresses.
As per stunner-host-config, the hosting Ziti tunneler is sending 3478/udp to 192.168.1.xxx address after exiting the Ziti network. That should be STUNner's IP, but from the successful call log, STUNner appears to be attached to a different LAN because it shows received packets from 10.0.0.4.
From STUNner's README:
Using STUNner a WebRTC deployment needs only two public-facing ports, one HTTPS port for signaling and a single UDP port for all your media.
This means the STUNner Ziti service's configs lack a TCP port for signaling over HTTPS.
22:22:27.043613 server.go:229: turn ERROR: Failed to handle datagram: failed to handle Send-indication from 10.0.0.4:59391: unable to handle send-indication, no permission added: 10.0.0.4:57071
This error from the unsuccessful log doesn't appear in the successful log. I'm guessing STUNner is enforcing one of its policies because it's complaining there's no permission for the client IP 10.0.0.4, which is the same IP that initiated the successful call. Did you reach any conclusions about that error? I'm unsure if it's significant.
Please confirm this is the desired system:
Two mobile devices running ZME and Matrix with permission to dial two services: stun.kincke.com acting as call broker and WebRTC over UDP relay, and a service that intercepts a wide private IP range.
Please confirm this is the desired (mal)function:
With ZME actively intercepting and either one of the two mobile devices on WiFi the Matrix call succeeds. However, if both mobile devices are on mobile data the callee is notified but the call never connects.
With the available info I'm guessing this means the wide range of private IPs in the second Ziti service is instrumental and no longer matches the peer address that is discovered by the STUN phase, i.e., STUNner is reporting a different, possibly public, peer IP that doesn't match the Ziti service.
Let me know if you get stuck on that because I have sample values for a separated management API with Traefik's Gateway mode (link to threadpost with two examples: Traefik TLS passthrough with Traefik CRDs and TLS passthrough with Gateway API). The Ziti controller chart's optional ingress values will not work with Traefik because Traefik doesn't support TLS passthrough for Ingress resources, and that's required for nearly all use cases involving Ziti's TLS listeners.
I did need to specify a different containerPort for the managementApi.
The winning combination was this:
managementApi:
# -- cluster service target port on the container
containerPort: 1281
# -- global DNS name by which routers can resolve a reachable IP for this service
advertisedHost: zac.domain.com
# -- cluster service, node port, load balancer, and ingress port
advertisedPort: 443
I used that with traefik's IngressRouteTCP CRD.
At this point, the only lingering thing is, how can I use my ClusterIssuer to have a letsencrypt certificate for the ZAC?
I did notice that I cannot access the ZAC via a ziti service, only via LAN.
I created a service pointing to the traefik IP address as I have done for all the rest of my services.
Is there something special that I need to do for the ZAC?
Also, is there a recommended way to have the ZAC automatically upgrade HTTP connections to HTTPS?
Yes. In summary, you successfully configured the controller to have separate container ports and Traefik TCP routes for the Ziti client and management APIs. Next you wish to configure a trusted cert for the Ziti console that is hosted on the same port as the management API.
This will make the trusted cert available on the management API port. The dnsNames must resolve to the external IP where you have published it with Traefik because Traefik must match the ClientHello server name on the request to route it to Ziti, and Ziti must match the server name too to present the correct certificate to your web browser.
As for upgrading to HTTPS, Ziti only listens with TLS and never without for the core control and edge and fabric ports, so you would need a load balancer configured to redirect HTTP requests to HTTPS.
This assumes you have declared a cluster issuer named like dns01-issuer-ziti-example-com. You could use an HTTP solver instead of a DNS solver if you are binding a trusted certificate for a public Traefik TCP route.
You navigated to the console after adding an alternative server cert spec to the controller's web identity, but you encountered the default web server cert instead of the new alt server cert you added to the web identity's input values.
Did you navigate to the console with the new name you specified for the alternative server cert, e.g., console.ziti.example.com?
The Ziti controller inspects the server name part of the TLS ClientHello when deciding which certificate to present to the client.
If that wasn't the cause, then let's double check that the templates rendered the required configmap. The controller's configmap has a list of "web" bindings. You should have at least two: one with edge-client (the client API) and one with edge-management (the mgmt API). They will have identical "identity" properties with a list of "alt_server_certs." I expect there's one item in that list with file paths that match the "mountPath" you specified in "webBindingPki." At that mountpoint you should find the contents of the TLS secret created by Cert Manager when the certificate was issued.
I described the ziti-controller-configConfigMap, and this is the web block:
web:
# name - required
# Provides a name for this listener, used for logging output. Not required to be unique, but is highly suggested.
- name: client
# bindPoints - required
# One or more bind points are required. A bind point specifies an interface (interface:port string) that defines
# where on the host machine the webListener will listen and the address (host:port) that should be used to
# publicly address the webListener(i.e. mydomain.com, localhost, 127.0.0.1). This public address may be used for
# incoming address resolution as well as used in responses in the API.
bindPoints:
#interface - required
# A host:port string on which network interface to listen on. 0.0.0.0 will listen on all interfaces
- interface: 0.0.0.0:1280
# address - required
# The public address that external incoming requests will be able to resolve. Used in request processing and
# response content that requires full host:port/path addresses.
address: ziti-controller.domain.com:8441
# Allows the webListener to have a specific identity instead of defaulting to the ctrl plane 'identity' section.
identity:
cert: /etc/ziti/web-client-identity/tls.crt
key: /etc/ziti/web-client-identity/tls.key
server_cert: /etc/ziti/web-identity/tls.crt
server_key: /etc/ziti/web-identity/tls.key
ca: /etc/ziti/web-identity/ca.crt
alt_server_certs:
- server_cert: "/etc/ziti/alt-server-cert/tls.crt"
server_key: "/etc/ziti/alt-server-cert/tls.key"
I also verified that the certificate is issued by letsencrypt with openssl x509 -in /etc/ziti/alt-server-cert/tls.crt -text -noout.
If you are still running the Ziti controller with separate ports for client and management APIs, then you will have at least one more web port (not 1280) that binds zac and edge-management. The controller chart re-uses the web PKI for both, so you should have an identical identity section in the other web binding.
Ensure there's an FQDN record like zac.domain.com resolving to Traefik's external IP, and a Traefik TCP route for requests like https://zac.domain.com/zac to ziti-controller-mgmt:443 (the mgmt service for the other web port).
You can probe the server certificate from outside the cluster to test the path used by the web browser.
The problem is that the alternative certificate's subject name, e.g., zac.domain.com, appears in both server certs, creating ambiguity during certificate selection.
The DNS SAN must be unique for the subject alternative name to ensure predictable certificate presentation.
To recap, you have two server certificates created by the controller chart:
the web PKI's server identity
the web PKI's alternative server identity
You must choose a new domain name for the alternative server identity that is distinct from the names that appear in the main server identity's list of DNS SANs.
