Need Guidance on Setting Up Multiple OpenZiti Routers and Identity Devices in EKS

1

I've set up OpenZiti in an EKS Cluster and successfully created an edge router. I connected an edge device (an EC2 instance) using ziti-edge-tunnel, and everything is functioning as expected.

Here’s a summary of the steps I followed:

YAML Files Used

I used the following YAML files to install the Controller and Router on the EKS Cluster via Helm:
Controller Values:

clientApi:
  advertisedHost: ziti-controller.example-domain.com
  advertisedPort: 443
  service:
    type: LoadBalancer
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        kubernetes.io/ingress.allow-http: "false"
        nginx.ingress.kubernetes.io/ssl-passthrough: "true"
ctrlPlane:
  advertisedHost: ziti-controller-ctrl.example-domain.com
  advertisedPort: 443
  service:
    enabled: true
    type: LoadBalancer
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        kubernetes.io/ingress.allow-http: "false"
        nginx.ingress.kubernetes.io/ssl-passthrough: "true"
persistence:
  enabled: true
  accessMode: ReadWriteOnce
  size: 10Gi
  storageClass: ebs-sc
cert-manager:
  enabled: true
  enableCertificateOwnerRef: true
  installCRDs: false
trust-manager:
  enabled: true
  app:
    trust:
      namespace: ziti-controller
  crds:
    enabled: false
ingress-nginx:
  enabled: true
  controller:
    extraArgs:
      enable-ssl-passthrough: "true"

Router Values:

ctrl:
  endpoint: ziti-controller-ctrl.example-domain.com:443
  advertisedHost: ziti-router.example-domain.com

edge:
  advertisedHost: ziti-router.example-domain.com
  advertisedPort: 443
  service:
    type: LoadBalancer
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        kubernetes.io/ingress.allow-http: "false"
        nginx.ingress.kubernetes.io/ssl-passthrough: "true"

linkListeners:
  transport:
    advertisedHost: ziti-router-transport.example-domain.com
    advertisedPort: 443
    service:
      enabled: true
      type: LoadBalancer
      ingress:
        enabled: true
        ingressClassName: nginx
        annotations:
          kubernetes.io/ingress.allow-http: "false"
          nginx.ingress.kubernetes.io/ssl-passthrough: "true"

persistence:
  enabled: true
  accessMode: ReadWriteOnce
  size: 1Gi
  storageClass: ebs-sc

Note: Domain names are hidden due to NDA. The domain example-domain.com is used as a placeholder.

Setting Up Multiple Routers
I need to set up multiple routers, generate three tokens, and then install the router on EKS three times. Here’s what I’ve done so far:

1. Create Three Enrollment Tokens for Routers in Three Zones

ziti edge create edge-router router-Zone-A \
  --role-attributes '#router-zone-A' \
  --tunneler-enabled \
  --jwt-output-file router-zone-A.jwt

ziti edge create edge-router router-Zone-B \
  --role-attributes '#router-zone-B' \
  --tunneler-enabled \
  --jwt-output-file router-zone-B.jwt

ziti edge create edge-router router-Zone-C \
  --role-attributes '#router-zone-C' \
  --tunneler-enabled \
  --jwt-output-file router-zone-C.jwt

2. Install Ziti Routers on EKS

helm install \
  --namespace ziti-router --create-namespace --generate-name \
  openziti/ziti-router \
    --set-file enrollmentJwt=router-zone-A.jwt \
    --values router-values-router-zone-A.yml

helm install \
  --namespace ziti-router --create-namespace --generate-name \
  openziti/ziti-router \
    --set-file enrollmentJwt=router-zone-B.jwt \
    --values router-values-router-zone-B.yml

helm install \
  --namespace ziti-router --create-namespace --generate-name \
  openziti/ziti-router \
    --set-file enrollmentJwt=router-zone-C.jwt \
    --values router-values-router-zone-C.yml

Setting Up Identity Devices
I also need to set up multiple identity devices, ensuring they connect only to their respective router. I used the following commands:

ziti edge create identity device EC2-1-Zone-A --role-attributes "#identity-zone-A" -o EC2-1-Zone-A.json
ziti edge create identity device EC2-2-Zone-A --role-attributes "#identity-zone-A" -o EC2-2-Zone-A.json
ziti edge create identity device EC2-3-Zone-A --role-attributes "#identity-zone-A" -o EC2-3-Zone-A.json

ziti edge create identity device EC2-1-Zone-B --role-attributes "#identity-zone-B" -o EC2-1-Zone-B.json
ziti edge create identity device EC2-2-Zone-B --role-attributes "#identity-zone-B" -o EC2-2-Zone-B.json
ziti edge create identity device EC2-3-Zone-B --role-attributes "#identity-zone-B" -o EC2-3-Zone-B.json

ziti edge create identity device EC2-1-Zone-C --role-attributes "#identity-zone-C" -o EC2-1-Zone-C.json
ziti edge create identity device EC2-2-Zone-C --role-attributes "#identity-zone-C" -o EC2-2-Zone-C.json
ziti edge create identity device EC2-3-Zone-C --role-attributes "#identity-zone-C" -o EC2-3-Zone-C.json

Router Policy Binding
I understand that I need to create a Router Policy to bind these identities to the appropriate routers. So that while connecting the tunnel they could connect to those routers only. I’ve used the following commands:

ziti edge create edge-router-policy zone-a-router-policy \
  --edge-router-roles "#router-zone-A" \
  --identity-roles "#identity-zone-A" \
  --semantic "AnyOf"

ziti edge create edge-router-policy zone-b-router-policy \
  --edge-router-roles "#router-zone-B" \
  --identity-roles "#identity-zone-B" \
  --semantic "AnyOf"

ziti edge create edge-router-policy zone-c-router-policy \
  --edge-router-roles "#router-zone-C" \
  --identity-roles "#identity-zone-B" \
  --semantic "AnyOf"

Questions

  1. Do I need to use the same router-values.yml file everywhere, or should I modify any values for each router?
  2. Will the identities with role attributes related to Zone-A connect only to the routers in Zone-A, and similarly for the other zones?

I’m new to this and would greatly appreciate any guidance or confirmation that I’m on the right track.

Thanks in advance!

2

You're on the right track and have a correct mental model overall. Nice going!

Thank you for the completeness and clarity of your post, too. That's helpful.

Your controller and router K8s Services will not use the Ingress Controller unless they have service.type: ClusterIP. That's mutually-exclusive to LoadBalancer, which will bypass ingress-nginx by asking the cloud provider to provision a separate public IP and load balancer for each K8s Service. One nice thing about using an Ingress Controller is that you only need one LoadBalancer service, which is typically a separate, billable item.

Changing the K8s Service type has ramifications for DNS and the controller and routers' advertised addresses, so let's adjust those carefully. The domain names that the controller and routers advertise must resolve to the public of the Ingress Controller's LoadBalancer Service, known by the value of "External IP" in the list of K8s Services (e.g., kubectl get services -n ingress-nginx). You may find it's suitable to configure a DNS * (wildcard) record for that public IP.

Your create command for the identities and routers has an error. Omit the # prefix when assigning roles to entities, and include it when granting roles on policies. Put another way, when assigning roles to an entity like an identity, service, or router, you must not prefix the role with #. Conversely, when granting roles on a policy, you must prefix roles with # to distinguish between roles and @ symbolic identifiers (e.g., @mention).

I know you're also exploring the restapi TF provider, so I should mention that when interfacing with the management API you must use the entity ID when granting to a single entity like @abcd1234 on a policy, whereas the ziti CLI will translate the friendly, symbolic entity name.

You can use a data source like this in your TF modules to resolve a symbolic entity name to an ID.

data "restapi_object" "identity22_lookup" {
    provider     = "restapi"
    path         = "/identities"
    search_key   = "name"
    search_value = "identity22"
}

example module

Some of the Helm values will be unique for each router. This is because each router is unique throughout the Ziti network. For example, any router that listens for connections must also advertise a distinct address.

A router can provide an edge listener for identities, a link listener for other routers, or both. Each listener is advertised by some address that must be reachable by its consumer.

Ensure you have a default router policy for services (SERP). This type of policy is rarely always needed (and rarely needs to be granular) and can be used to constrain which routers handle a service's traffic. Your router policies for identities (ERP) will ensure that the services are dialed and bound by the authorized identities.

ziti edge create service-edge-router-policy "default" \
    --edge-router-roles '#all' --service-roles '#all'
1 Like
3

Thanks for your response. I had set up the following policy earlier:

ziti edge create service-edge-router-policy "default" \
    --edge-router-roles '#all' --service-roles '#all'

However, this was interfering with other policies. Specifically, I wanted identities with the attribute identity-zone-A to connect (tunnel) only to the router in Zone A. After removing this default policy, everything worked as expected, and now the tunnels are connecting to their respective routers as intended.

Unfortunately, we won’t be using Terraform for this part of the process—Terraform is currently only being used to set up the cluster and ArgoCD. The installations and configurations will be managed through ArgoCD.

I’m planning to set up GitHub Actions workflows to automate tasks like managing identities, policies, etc. This way, any changes made in the repo will automatically update the corresponding configurations.

4

Cool. The router policies for services (SERP) can only limit, not expand, the set of routers available to an identity-service pair.

If the router policies for identities (ERP) allow A-to-A, then a permissive SERP has no effect.

I wonder if overriding the default ERP semantic was a factor, but I don't see how it could be if the samples you shared represent three discrete zones, A, B, and C, with no overlapping policies and no entities having roles for more than one zone.

1 Like
5

Thank you for the insight! Yes, I did change the ERP semantic from "AnyOf" to "AllOf" to ensure that identities strictly connect to their corresponding routers (A-to-A, B-to-B, etc.). I understand now that with discrete zones and no overlapping roles, the change might not have been strictly necessary. However, it did help in making sure the connections behaved as expected. I appreciate your input—it's helped clarify the interplay between ERP and SERP in my setup.

1 Like