Ziti proxy tunnel mode in k8s

1

Hello, i'm running for a while a ziti overlay for my home network, and it works really nice, kudos for the amazing work :tada:.

As it is right now, i have a ziti router and a ziti controller running on azure, a couple of ziti desktop edge clients and a ziti-host container running on a Raspberry Pi.

I am trying to setup a HA k8s cluster on 3 Raspberry PIs, and i want to integrate it in the ziti overlay. (i'm new both to k8s and ziti).

From what i understand i would just need to add a router to the cluster, and i should be good(easier said than done :sweat_smile:).

I would like to setup the cluster in such a way that all "public" access should be thru the overlay, and the access granularity should be at the k8s service level(So ideally i would like that ziti identity A and B to have access to k8s service 1, and ziti identity C to have access to k8s service 2). The cluster is setup on a different vnet compared to the rest of the devices on my LAN and traffic is blocked by firewall between the vnets.

Looking at the documentation here it sounds like this is exactly what i want.

I have a couple of questions:

1 Is there more documentation/examples on how to set the k8s router in proxy mode? i get the following error on my first try to set this up: error: identity is not valid for provided host: [router-edge.openziti.svc]. is valid for: [, 127.0.0.1, localhost]]","time":"2025-04-24T21:30:19.520Z"

2 If i use the proxy mode this implies that this will route all the traffic thru dns(CoreDNS). Do i need to add an ingress to the cluster(from my understanding as all the k8s services will be of ClusterIP type and they will be "exposed" thru the ziti proxy router, there is no need for ingress)?

3 As the ziti router will be private, and the cluster blocked by firewall from there rest of the LAN, this will imply that all the traffic is gonna be routed to thru my public ziti router on azure, is this assumption correct? I was thinking to add one more router on the LAN that has access to both home and k8s vnet so the devices can be routed on LAN. I don't care much about discovery, but if i will have heavy traffic between devices on LAN i don't want that to go thru azure.

4 IF i get the documentation right, when i configure the router in proxy mode it will create the ziti services. Based on this assumption i was thinking to do the RBAC outside of k8s(creating the 2 configs and 2 service policies per service and associated the identities/attributes). Is this a good idea for managing the overlay?

2

Let me clarify something before answering the rest. Will pods be connecting to Ziti services from inside the Kubernetes cluster, or will you only publish existing ClusterIP services with Ziti?

3

i was thinking to just expose the services. I am still in the early stage of the project so i'm quite flexible in what is the end outcome. Reading the documentation this felt like something that would meet my requirements and somewhat simple to achieve.
My main reason why i was thinking of doing it at the service level is that k8s will ensure the fault tolerance as i'm running a HA cluster.
If you think there is a better way of achieving this i'm all ears(or eyes in this case😅)

1 Like
4

OK, here's an overview of publishing ClusterIP services with Ziti when none of the Ziti clients are pods inside the same cluster.

  1. Publish the ClusterIP services, noting their cluster-internal domain name and port
  2. For each ClusterIP service, create a Ziti service with a host.v1 configuration targeting the cluster-internal domain name and port. Suppose the clients will access the Ziti service through a Ziti tunneler (i.e., a local transparent proxy). In that case, you also need an intercept.v1 config for each service specifying some address:port, which can be the same or different from the target address:port.
  3. Since your Cluster is HA, I think you care about resilience and will recommend hosting your services with a ziti router deployment.
  4. In Ziti, create an edge router with the tunneling-enabled flag for each cluster where you will host services. If your services are partitioned by network policy you'll need one Ziti router per partition.
  5. Define Bind Service Policies for each group of routers that will host a set of services, e.g., #my-k8s-cluster-ziti-service-hosts are granted Bind permission for #my-k8s-cluster-services
  6. Deploy the ziti-router Helm chart.
  7. When the counterpart Ziti identity appears in Ziti after deploying the router, grant the identity a role that matches your Bind Service Policy for the appropriate Ziti services you wish that router to provide, e.g., #my-k8s-cluster-ziti-service-hosts.
5

But is my understanding correct do i need to run it in proxy mode?`

6

No, not if this is accurate :backhand_index_pointing_up:

7

The proxy mode for the Ziti router's tunneler is used to make external services available via Ziti to applications running inside your cluster (the pods are the clients of the application).

Your goal is different – you want to make internal ClusterIP services available to clients outside the cluster with Ziti (the pods are the application servers).

The proxy mode handles the reverse traffic flow compared to what you need.

1 Like
8

i've gave it another try, right now i'm providing the following values to the helm chart:

ctrl:
  endpoint: <ctrl-endpoint>
linkListeners:
  transport:
    enabled: false
persistance:
  enabled: true
  existingClaim: router
  size: 50Mi
  storageClass: csi-nfs
enrollmentJwt: |
  <jwt>
image:
  additionalArgs:
    - --verbose

but the router wouldn't start properly, i can see in zac the the router gets verified, but it never gets online. The pod logs look like this:

INFO: config file exists in /etc/ziti/config/ziti-router.yaml
{"file":"github.com/openziti/ziti/ziti/util/updates.go:84","func":"github.com/openziti/ziti/ziti/util.LogReleaseVersionCheck","level":"debug","msg":"ZITI_CHECK_VERSION is not 'true'. skipping version check","time":"2025-05-10T20:42:27.580Z"}
{"file":"github.com/openziti/ziti/router/configedge.go:394","func":"github.com/openziti/ziti/router.(*EdgeConfig).loadCsr","level":"info","msg":"loaded csr info from configuration file at path [edge.csr]","time":"2025-05-10T20:42:27.593Z"}
{"file":"github.com/openziti/ziti/router/configedge.go:138","func":"github.com/openziti/ziti/router.(*EdgeConfig).LoadEdgeConfigFromMap","level":"debug","msg":"cached data model file set to: /etc/ziti/config/ziti-router.yaml.proto.gzip","time":"2025-05-10T20:42:27.593Z"}
{"file":"github.com/openziti/ziti/router/configedge.go:155","func":"github.com/openziti/ziti/router.(*EdgeConfig).LoadEdgeConfigFromMap","level":"warning","msg":"Invalid heartbeat interval [0] (min: 60, max: 10), setting to default [60]","time":"2025-05-10T20:42:27.593Z"}
{"file":"github.com/openziti/ziti/router/configedge.go:325","func":"github.com/openziti/ziti/router.parseEdgeListenerOptions","level":"info","msg":"advertised port [0] in [listeners[443].options.advertise] does not match the listening port [0] in [listeners[3022].address].","time":"2025-05-10T20:42:27.593Z"}
{"file":"github.com/openziti/ziti/router/config.go:918","func":"github.com/openziti/ziti/router.LoadConfigWithOptions","level":"fatal","msg":"one or more advertise addresses are invalid: [invalid listeners.binding.advertise: router-edge.openziti.svc:443, error: identity is not valid for provided host: [router-edge.openziti.svc]. is valid for: [, 127.0.0.1, localhost]]","time":"2025-05-10T20:42:27.593Z"}

Looking at the logs i assume is should provide the service domain( router-edge.openziti.svc) as an additional value somewhere, going thru zac or the value reference for the helm chart, i couldn't find a place, i tried to add it to the edge.advertisedHost but that didn't make a difference.
What am i doing wrong?

9

@qrkourier do you have any idea why this error is popping up?

10

You encountered the router's startup check that ensures it's only advertising DNS names for which it has DNS SANs on its certificates.

I know there was one Helm chart change in the last few months that addressed a possible configuration leading to the same error. Did you already verify there are no newer Helm chart versions available?

helm repo update openziti
helm search repo openziti/ziti-router
helm list -n openziti

The router will try to obtain a new certificate for the configured DNS names when it starts, but if it fails to start because it already has a mismatch then it may be easiest to re-enroll or replace the router.

If you have Ziti router policies tied to the router by explicit @mention, then you should re-enroll because replacing the router will break those policies.

The procedure to re-enroll a router in Kubernetes involves:

  1. get a new enrollment token

    ziti edge re-enroll edge-router router1 --jwt-output-file /tmp/router1.jwt

  2. feed it to the Helm release with an additional var to force the enrollment step

    helm get values ziti-router --all | tee /tmp/ziti-router.yaml
helm upgrade ziti-router openziti/ziti-router --values /tmp/ziti-router.yaml --set bootstrapEnv.ZITI_BOOTSTRAP_ENROLLMENT=force --set-file enrollmentJwt=/tmp/router1.jwt