Regarding certificate verification for management operations with the CLI, including ziti edge login: the CLI trusts the default management API certificate through the trust-on-first-use (TOFU) mechanism built into ziti edge login by prompting the user to accept the certificate and storing it in ~/config/ziti/. Alternatively to TOFU, the root CA for the management API's certificate can be explicitly trusted with the ziti edge login --ca option.
It's unclear why the stored password isn't valid. I'm not aware of any conditions that could lead to this situation. Maybe an incorrect database snapshot was restored?
If you have only one admin identity and that identity has only password authentication enabled and you do not have the correct password then your only option is to perform a database recovery operation after adding a new admin user. Here's the procedure. It's long and, hopefully, precise enough. If this works well for you I'll develop this draft as documentation for the ops guides.
Add a new identity with admin privilege, substituting your desired username for ZITI_ADMIN_USER and the desired password for ZITI_ADMIN_PASSWORD. This username must be unique. Optionally, you may later use the new admin's credential to reset the default admin's password.
kubectl get pods --selector app.kubernetes.io/name=ziti-controller --output jsonpath="{.items[0].metadata.name}" \
| xargs -IPOD kubectl -c ziti-controller exec POD -- \
ziti ops db add-debug-admin /persistent/ctrl.db-20241022-154414 ZITI_ADMIN_USER ZITI_ADMIN_PASSWORD
{"file":"github.com/openziti/storage@v0.3.2/boltz/migration.go:99","func":"github.com/openziti/ziti/controller/db.RunMigrations.(*migrationManager).Migrate.func1","level":"info","msg":"edge datastore is up to date at version 37","time":"2024-10-22T15:46:59.792Z"}
You initialized a new controller database with a new default admin password, right?
That confirms the Ziti network is functioning, but the router is not hosting the assigned service for some reason.
A quick review of the thread reminds me that we confirmed the service edge router policy (SERP) was permissive (#all/#all), but some time has passed and the router error you reported "failed to dial fabric" is precisely the error I expect to see if the hosting or dialing router's tunnler has not been granted permission to use the service by a SERP.
The initial symptom was "no terminators" and that's been solved by deploying a second tunneler with permission to host/bind the service, but the router may still be logging "failed to dial fabric," which would allow us to investigate the root cause. A router's tunneler certainly will host a service if it has permission!
Things to check:
SERP is all/all
policy-advisor (as before) shows the router's tunneler has bind permission
the router's tunneler was enabled when it was created or subsequently updated to enable the tunneler and it has a tunnel binding mode "host" in its configuration
Here's an overview of creating a Ziti service. You wish to map a fictitious domain name to a real IP address, so I assume you will use a Ziti tunneler on both ends: intercept (client) and host (server).
Create an intercept.v1 config. This defines the fictitious domain names that will exist in the client tunneler's nameserver.
Create a host.v1 config. This defines the address of the target server as a domain name or IP address, port, protocol.
Create a service with the two configs above with a role like "acme-services."
Create a Dial service policy with roles like "acme-clients"/"acme-services." This grants client tunneler identities permission to dial the service.
Create a Bind service policy with roles like "acme-hosts"/"acme-services." This grants hosting tunneler identities permission to bind (host) the service.
Create some client identities with role "acme-clients."
Create some hosting identities with role "acme-hosts."
Deploy the client identities on devices where "service.domain.com" should be intercepted.
Deploy the hosting identities on devices that have access to the target address from the host.v1 config.
You can extend this pattern by adding more services with role "acme-services" if the service should be dialed and hosted by the same set of identities.
Run ziti edge update identity "edge-router" to grant whichever role you have granted the ziti-host identity from the bind service policy
It should be host to enable the router's tunneler to host services. It was default none because the router panics and won't start if not tunneler enabled when created. You can enable it later with ziti edge update edge-router "edge-router" --tunneler-enabled.
Ah.
Swapping that value from none to host allowed everything to work with the default router
I think I figured out why video and voice calls are failing in matrix, but I do not have a solution to it yet.
Matrix does 1:1 calls peer-to-peer so they are end-to-end encrypted.
With a traditional VPN, both peers are on the same network, so they can connect.
Given that matrix is isolated with OpenZiti, the peer-to-peer connection fails.
I played with Jitsi meet and zrok and found that it functions over a single TCP port (link to blog post with example), so it's doable with a zrok share (which uses OpenZiti under the hood), but I didn't explore how well it would perform with more than two parties or other things that would strain the resources.
We really need to understand the P2P requirements in terms of directional network flows, e.g., proto://src_host:src_port => proto://dst_host:dst_port, like a firewall rule.
Once we know that we can probably identify a combination of Ziti service configs that will work.
Are you mainly trying to get the 1:1 Matrix calls working vs. group/conference?
I tried splitting the management service by setting managementApi.service.enabled to true in the controller chart.
I configured traefik as the ingressClassName, and set the managementApi.advertisedHost as well.
If I leave the advertisedPort as 443, the controller logs show this when I try to access the console (at https://zac.domain.com):
{"_context":"tls:0.0.0.0:1281","error":"remote error: tls: bad certificate","file":"github.com/openziti/transport/v2@v2.0.143/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.160:43488","time":"2024-10-30T02:23:57.483Z"}
If I change the advertisedPort to 80, the logs show this:
{"_context":"tls:0.0.0.0:1281","error":"tls: first record does not look like a TLS handshake","file":"github.com/openziti/transport/v2@v2.0.143/tls/listener.go:257","func":"github.com/openziti/transport/v2/tls.(*sharedListener).processConn","level":"error","msg":"handshake failed","remote":"10.0.0.160:47994","time":"2024-10-30T02:25:13.615Z"}
Learn how Matrix 1:1 calls perform client exchange and write Ziti service configs that match. For example, if Matrix assumes the client's private IP address is reachable by peers, and presents that address to the other party to connect the WebRTC stream, then a Ziti service intercept config is needed for each party's private IP to make them both reachable by Ziti peers.
Configure Matrix to use a Ziti intercept as client ID. This may require modifying or extending the Matrix software if it's not configurable.
Host a TURN server to relay the WebRTC stream. Either the TURN server is public or reachable via Ziti for both parties.
As you expected, this boolean triggers logic in the controller chart to split the client and management APIs to separate web listeners so you can control access. By default, the management operations are available in the client API's web listener.
Your input values should resemble these:
managementApi:
advertisedHost: zac.domain.com # any domain name that resolves to Traefik external IP; distinct from clientApi
advertisedPort: 443 # where clients will connect to Traefik Ingress
containerPort: 1281 # any available container port; distinct from clientApi
ingress:
annotations: {} # any annotations required to configure Traefik Ingress controller for passthrough TLS
enabled: true
ingressClassName: traefik
service:
enabled: true
type: ClusterIP
The URL for the console with the advertised host and port you configured is printed when you upgrade the Helm release with the new input values.
❯ helm -n ziti upgrade ziti-controller openziti/ziti-controller --values ziti-controller.yaml
Release "ziti-controller" has been upgraded. Happy Helming!
NAME: ziti-controller
LAST DEPLOYED: Wed Oct 30 11:47:38 2024
NAMESPACE: ziti
STATUS: deployed
REVISION: 10
TEST SUITE: None
NOTES:
Your release ziti-controller was upgraded.
You have chart version 1.1.1 and app version 1.1.15.
To learn more about the release, try:
$ helm status ziti-controller -n ziti
$ helm get all ziti-controller -n ziti
This deployment provides an OpenZiti controller to manage an OpenZiti network.
Visit the console in a web browser: https://zac.domain.com:443/zac/
