Troubleshooting OpenZiti Tunnel: Edge Router and Service Configuration Issues

am3y · December 28, 2024, 12:09am

Disclaimer:

Due to a non-disclosure agreement (NDA), the details in this post have been modified and are based on a hypothetical example for illustrative purposes. The core issue, however, remains representative of the actual challenge.

Scenario:

Two VPCs are configured: VPC-public and VPC-private. Note: These public, private are just names for our convinence.
Two EC2 instances are present:
- EC2-public in VPC-public.
- EC2-private in VPC-private.
An EKS cluster is set up in VPC-private.
I have also posted a simple diagram to understand the architecture

The following OpenZiti components have been deployed:

Ziti Controller

The controller was deployed with the following values.yml configuration:

clientApi:
  advertisedHost: ziti-controller.example.com
  service:
    enabled: true
    type: ClusterIP

  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      external-dns.alpha.kubernetes.io/hostname: "ziti-controller.example.com"
      service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # Ensures the LB is public


ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP




highAvailability:
  # -- Ziti controller HA mode
  mode: standalone
  # -- Ziti controller HA swarm replicas
  replicas: 1


persistence:
  enabled: true
  storageClass: "ebs-sc"
  accessMode: ReadWriteOnce
  size: 3Gi


cert-manager:
  enabled: true
  enableCertificateOwnerRef: true
  installCRDs: false

trust-manager:
  enabled: true
  app:
    trust:
      namespace: "ziti-controller"
  crds:
    enabled: false

ingress-nginx:
  enabled: true
  controller:
    extraArgs:
      enable-ssl-passthrough: "true"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"

Ziti Routers

Two routers were set up:

Router-Private:

 ctrl:
   endpoint: ziti-controller.example.com:443
   advertisedHost: ziti-router-private.example.com

 # Edge configuration for external identities
 edge:
   advertisedHost: ziti-router-private.example.com
   advertisedPort: 443
   service:
     type: LoadBalancer  
     annotations:
       external-dns.alpha.kubernetes.io/hostname: ziti-router-private.example.com
       service.beta.kubernetes.io/aws-load-balancer-internal: "true" # loadbalancer here is private
   ingress:
     enabled: false

 # Link listeners for router-to-router communication (internal)
 linkListeners:
   transport:
     advertisedHost: ziti-router-transport-private.example.com
     advertisedPort: 443
     service:
       enabled: true
       type: ClusterIP  # All routers are internal; no external exposure
     ingress:
       enabled: false  # Not needed as routers are internal

 # Persistence for router data
 persistence:
   enabled: true
   accessMode: ReadWriteOnce
   size: 1Gi
   storageClass: ebs-sc

Router-Public:

 ctrl:
   endpoint: ziti-controller.example.com:443
   advertisedHost: ziti-router-public.example.com

 # Edge configuration for external identities
 edge:
   advertisedHost: ziti-router-public.example.com
   advertisedPort: 443
   service:
     type: LoadBalancer  
     annotations:
       external-dns.alpha.kubernetes.io/hostname: ziti-router-public.example.com
       service.beta.kubernetes.io/aws-load-balancer-internal: "false" # Loadbalancer here is public
   ingress:
     enabled: false

 # Link listeners for router-to-router communication (internal)
 linkListeners:
   transport:
     advertisedHost: ziti-router-transport-public.example.com
     advertisedPort: 443
     service:
       enabled: true
       type: ClusterIP  # All routers are internal; no external exposure
     ingress:
       enabled: false  # Not needed as routers are internal

 # Persistence for router data
 persistence:
   enabled: true
   accessMode: ReadWriteOnce
   size: 1Gi
   storageClass: ebs-sc

The routers were installed using Helm with enrollment JWTs:

helm install ziti-router-private-release \
  --namespace ziti-router --create-namespace \
  openziti/ziti-router \
  --set-file enrollmentJwt=router-private.jwt \
  --values router-values-private.yml

helm install ziti-router-public-release \
  --namespace ziti-router --create-namespace \
  openziti/ziti-router \
  --set-file enrollmentJwt=router-public.jwt \
  --values router-values-public.yml

Ziti Edge Tunnel Configuration

The identity for EC2-public was created, enrolled, and the Ziti tunnel was started:

ziti edge create identity device EC2-public --role-attributes "EC2-public" -o EC2-public.jwt
ziti edge enroll EC2-public.jwt -o EC2-public.json

The identity for EC2-private was created, enrolled, and the Ziti tunnel was started:

ziti edge create identity device EC2-private --role-attributes "EC2-private" -o EC2-private.jwt
ziti edge enroll EC2-private.jwt -o EC2-private.json

Edge router policies were created to bind each EC2 instance to its respective router:

ziti edge create edge-router-policy router-private-router-policy \
  --edge-router-roles "#router-private" \
  --identity-roles "#EC2-private" \
  --semantic "AllOf"

ziti edge create edge-router-policy router-public-router-policy \
  --edge-router-roles "#router-public" \
  --identity-roles "#EC2-public" \
  --semantic "AllOf"

Our Task

So in EC2-public we have a application which is accessed on TCP:80. We want to access that application from EC2-private. For testing we have installed apache2 where we can just run the curl command to test.

So to achieve this what I ran the below commands:

Intercept Configuration

ziti edge create config apache-intercept-ip intercept.v1 '{
   "protocols": ["tcp"],
   "addresses": ["10.100.99.99"],
   "portRanges": [{"low": 80, "high": 80}]
}'

Host Configuration

ziti edge create config apache-host.v1 host.v1 '{
   "protocol": "tcp",
   "address": "127.0.0.1",
   "port": 80
}'

Create the Service

ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1

Create Service Policies

Dial Policy: Allow EC2-private to access the service:

ziti edge create service-policy apache-dial-policy Dial --service-roles "@apache-service" --identity-roles "#EC2-private"

Bind Policy: Allow EC2-public to host the service:

ziti edge create service-policy apache-bind-policy Bind --service-roles "@apache-service" --identity-roles "#EC2-public"

Connecting the tunnels:

ziti-edge-tunnel run -i EC2-public.json
ziti-edge-tunnel run -i EC2-private.json

Now both the tunnels get connected, technically I have to run curl http://10.100.99.99 to test the connection here. But it is not connecting.
After checking the tunnels logs I have found that there is some issue in the EC2-public

Logs and Outputs

ziti-edge-tunnel logs from EC2-public:

About to run tunnel service... ziti-edge-tunnel
(7088)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(7088)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2024-12-27T23:55:31.751)
(7088)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(7088)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(7088)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(7088)[        0.011]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(7088)[        0.011]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-public.json] is not loaded yet or already removed.
(7088)[        0.011]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-public.json]
(7088)[        0.011]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-public.json]
(7088)[        0.011]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-public.json] loaded
(7088)[        0.012]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(7088)[        0.012]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(7088)[        0.012]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(7088)[        0.012]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(7088)[        0.072]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(7088)[        0.097]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:798 on_event() ztx[EC2-public.json] API Event with controller address : (null)
(7088)[        0.100]   ERROR tunnel-cbs:ziti_tunnel_ctrl.c:1571 update_config_done() updated config file with new URL
(7088)[        0.102]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-public] connected to controller
(7088)[        0.102]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-public.json] context event : status is OK
(7088)[        0.130]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-public) new channel for ztx[0] identity[EC2-public]
(7088)[        0.130]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-public] added edge router router-public@ziti-router-public.example.com
(7088)[        0.130]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(7088)[        0.212]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:953 on_service() hosting server_address[tcp:127.0.0.1:80] service[apache-service]
(7088)[        0.212]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:1Yq8uc7EIO0FYT6dD2311q ===============
(7088)[        0.212]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
(7088)[        0.216]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
(7088)[        0.218]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(7088)[        0.239]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(7088)[        0.239]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-public] router router-public connected
(7088)[        0.239]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(7088)[        1.096]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(7088)[        2.130]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(7088)[        3.374]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(7088)[        7.639]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE

ziti-edge-tunnel logs from EC2-private:

About to run tunnel service... ziti-edge-tunnel
(8197)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(8197)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2024-12-27T23:55:39.222)
(8197)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(8197)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(8197)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(8197)[        0.019]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-private.json] is not loaded yet or already removed.
(8197)[        0.019]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-private.json]
(8197)[        0.019]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-private.json]
(8197)[        0.019]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-private.json] loaded
(8197)[        0.020]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(8197)[        0.020]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(8197)[        0.020]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(8197)[        0.020]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(8197)[        0.020]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(8197)[        0.081]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(8197)[        0.109]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:798 on_event() ztx[EC2-private.json] API Event with controller address : (null)
(8197)[        0.111]   ERROR tunnel-cbs:ziti_tunnel_ctrl.c:1571 update_config_done() updated config file with new URL
(8197)[        0.113]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-private] connected to controller
(8197)[        0.113]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-private.json] context event : status is OK
(8197)[        0.140]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-private) new channel for ztx[0] identity[EC2-private]
(8197)[        0.140]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-private] added edge router router-private@ziti-router-private.example.com
(8197)[        0.140]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(8197)[        0.343]    INFO tunnel-cbs:ziti_tunnel_cbs.c:414 new_ziti_intercept() creating intercept for service[apache-service] with intercept.v1 = { "addresses": [ "10.100.99.99" ], "portRanges": [ { "high": 80, "low": 80 } ], "protocols": [ "tcp" ] }
(8197)[        0.343]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:938 on_service() starting intercepting for service[apache-service]
(8197)[        0.343]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:1Yq8uc7EIO0FYT6dD2311q ===============
(8197)[        0.343]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 2 route updates
(8197)[        0.347]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[2]: 0/OK
(8197)[        0.373]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(8197)[        0.373]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-private] router router-private connected
(8197)[        0.373]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(8197)[        1.107]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected

Any guidance or insights to resolve this issue would be greatly appreciated. Thank you!

scareything · December 29, 2024, 10:34pm

This looks like the identity is not allowed to use any edge routers. Have you created any edge router policies?

You can get a quick summary of your overall policy setup with ziti edge policy-advisor. Does it show any issues?

am3y · December 30, 2024, 12:09am

Thanks for your reply @scareything

Here is the output of that command

$ ziti edge policy-advisor services

Policy General Guidelines
  In order for an identity to dial or bind a service, the following must be true:
    - The identity must have access to the service via a service policy of the correct type (dial or bind)
    - The identity must have access to at least one on-line edge router via an edge router policy
    - The service must have access to at least one on-line edge router via a service edge router policy
    - There must be at least one on-line edge router that both the identity and service have access to.

Policy Advisor Output Guide:
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR. 
  ID = identity name
  ID ROUTERS = number of routers accessible to the identity via edge router policies.
    - See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
  SVC = service name
  SVC ROUTERS = number of routers accessible to the service via service edge router policies.
    - See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
  ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
  COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
  DIAL_OK = indicates if the identity has permission to dial the service.
    - See service polices for a service  : ziti edge controller list service service-policies <service>
    - See service polices for an identity: ziti edge controller list identity service-policies <identity>
  BIND_OK = indicates if the identity has permission to bind the service.
  ERROR_LIST = if the status is ERROR, error details will be listed on the following lines

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
ERROR: EC2-private (1) -> apache-service (0) Common Routers: (0/0) Dial: Y Bind: N 
  - Service has no edge routers assigned. Adjust service edge router policies.

ERROR: EC2-public (1) -> apache-service (0) Common Routers: (0/0) Dial: N Bind: Y 
  - Service has no edge routers assigned. Adjust service edge router policies.

$ ziti edge policy-advisor identities

Policy General Guidelines
  In order for an identity to dial or bind a service, the following must be true:
    - The identity must have access to the service via a service policy of the correct type (dial or bind)
    - The identity must have access to at least one on-line edge router via an edge router policy
    - The service must have access to at least one on-line edge router via a service edge router policy
    - There must be at least one on-line edge router that both the identity and service have access to.

Policy Advisor Output Guide:
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR. 
  ID = identity name
  ID ROUTERS = number of routers accessible to the identity via edge router policies.
    - See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
  SVC = service name
  SVC ROUTERS = number of routers accessible to the service via service edge router policies.
    - See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
  ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
  COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
  DIAL_OK = indicates if the identity has permission to dial the service.
    - See service polices for a service  : ziti edge controller list service service-policies <service>
    - See service polices for an identity: ziti edge controller list identity service-policies <identity>
  BIND_OK = indicates if the identity has permission to bind the service.
  ERROR_LIST = if the status is ERROR, error details will be listed on the following lines

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
ERROR: EC2-private (1) -> apache-service (0) Common Routers: (0/0) Dial: Y Bind: N 
  - Service has no edge routers assigned. Adjust service edge router policies.

ERROR: EC2-public (1) -> apache-service (0) Common Routers: (0/0) Dial: N Bind: Y 
  - Service has no edge routers assigned. Adjust service edge router policies.

ERROR: Default Admin 
  - Identity does not have access to any services. Adjust service policies.

ERROR: router-private 
  - Identity does not have access to any services. Adjust service policies.

ERROR: router-public 
  - Identity does not have access to any services. Adjust service policies.

scareything · December 30, 2024, 11:54am

Similar to how you can restrict access to services based on identities or attributes, OpenZiti also allows you to limit which services an edge router may handle connections for, and which identities may use the edge router. Edge router access is controlled with service-edge-router-policy and edge-router-policy. A special roll #all may be used here if you want to allow edge routers to be used for all services and/or by all identities:

ziti edge create service-edge-router-policy all-services-on-all-routers --edge-router-roles '#all' --service-roles '#all'

ziti edge create edge-router-policy all-identities-on-all-routers --edge-router-roles '#all' --identity-roles '#all'

Of course you can use more specific identity/service/router roles as your situation requires.

am3y · January 2, 2025, 9:13pm

@scareything Thank you so much for your detailed response and guidance! I appreciate the time and effort you've put into helping me troubleshoot this issue.

I followed your advice and tested with the following commands:

ziti edge create service-edge-router-policy all-services-on-all-routers \
      --edge-router-roles '#all' \
      --service-roles '#all'

ziti edge create edge-router-policy all-identities-on-all-routers \
      --edge-router-roles '#all' \
      --identity-roles '#all'

With these broad policies, I was able to successfully test the service using:

curl http://10.100.99.99/

Setting Up Specific Routers

Since I wanted to restrict specific routers to handle traffic based on their usage, I created a new router named router-services with the following configuration:

ctrl:
  endpoint: ziti-controller.example.com:443
  advertisedHost: ziti-router-services.ziti-router.svc.cluster.local

# Edge configuration for external identities
edge:
  advertisedHost: ziti-router-services.ziti-router.svc.cluster.local
  advertisedPort: 443
  service:
    type: ClusterIP
  ingress:
    enabled: false

# Link listeners for router-to-router communication (internal)
linkListeners:
  transport:
    advertisedHost: ziti-router-transport-services.ziti-router.svc.cluster.local
    advertisedPort: 443
    service:
      enabled: true
      type: ClusterIP
    ingress:
      enabled: false

Additionally, I applied the following edge-router policies to restrict EC2 instances to their respective routers:

ziti edge create edge-router-policy router-private-router-policy \
--edge-router-roles "#router-private" \
--identity-roles "#EC2-private" \
--semantic "AllOf"

ziti edge create edge-router-policy router-public-router-policy \
--edge-router-roles "#router-public" \
--identity-roles "#EC2-public" \
--semantic "AllOf"

I set up the service and policies as follows:

ziti edge create config apache-intercept-ip intercept.v1 '{
    "protocols": ["tcp"],
    "addresses": ["10.100.99.99"],
    "portRanges": [{"low": 80, "high": 80}]
}'

ziti edge create config apache-host.v1 host.v1 '{
    "protocol": "tcp",
    "address": "127.0.0.1",
    "port": 80
}'

ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1

ziti edge create service-policy apache-dial-policy Dial \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-private"

ziti edge create service-policy apache-bind-policy Bind \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-public"

Test Cases and Issues

CASE-1: Specific Router Services Policy

I used this command to configure the service to use router-services:

ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
      --edge-router-roles '#router-services' \
      --service-roles '#apache-service'

On the EC2-public screen session, I observed the following repeated error:

WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE

In the EC2-private screen session, no errors were logged.

CASE-2: Broad Identity Policy

I reverted to the following policy:

ziti edge create edge-router-policy all-identities-on-all-routers \
      --edge-router-roles '#all' \
      --identity-roles '#all'

This caused the following error in the EC2-private screen session:

ERROR ziti-sdk:connect.c:1071 connect_reply_cb() conn[0.0/xxeoGuM2/Connecting](apache-service) failed to connect, reason=can't route from SrmOpTMM28 -> RhfQp6BBU8

Current Setup

Here is the output of the edge routers:

ziti edge list edge-routers
╭────────────┬─────────────────┬────────┬───────────────┬──────┬─────────────────╮
│ ID         │ NAME            │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES      │
├────────────┼─────────────────┼────────┼───────────────┼──────┼─────────────────┤
│ 6ofQaTMMU8 │ router-services │ true   │ true          │    0 │ router-services │
│ RhfQp6BBU8 │ router-public   │ true   │ true          │    0 │ router-public   │
│ SrmOpTMM28 │ router-private  │ true   │ true          │    0 │ router-private  │
╰────────────┴─────────────────┴────────┴───────────────┴──────┴─────────────────╯

Questions

Router-Services Configuration:
- Is the ClusterIP configuration for router-services causing any limitations? Should I update the advertisedHost in router-services?
Service Policies:
- Are the service-edge-router policies for router-services correctly set? Should additional configurations be applied?
Routing Errors:
- In CASE-2, the error mentions, can't route from SrmOpTMM28 -> RhfQp6BBU8. Does this indicate a misconfiguration in edge-router or service-edge-router policies?

Could you help identify what’s causing these issues and suggest any changes? Thank you in advance!

scareything · January 3, 2025, 3:13pm

The name of your service is apache-service, but you are specifying an attribute (with '#') when you create the service edge router policy:

And from what I can see you didn't assign any attributes to that service. So if I'm right about that then you could either add the attribute to your service:

ziti edge update service apache-service --role-attributes apache-service

You may have a reason for doing this, but it seems redundant to me at first glance. I'd suggest not creating attributes that mirror the names of your entities, and using the entity names directly (by prefixing with '@') when the intent is to specify a specific identity/service/router. For example:

ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
      --edge-router-roles '#router-services' \
      --service-roles '@apache-service'

am3y · January 3, 2025, 4:22pm

Thanks for your speedy reply @scareything . I tried the following steps as you suggested:

ziti edge create edge-router-policy router-private-router-policy \
--edge-router-roles "#router-private" \
--identity-roles "#EC2-private" \
--semantic "AllOf"

ziti edge create edge-router-policy router-public-router-policy \
--edge-router-roles "#router-public" \
--identity-roles "#EC2-public" \
--semantic "AllOf"

ziti edge create config apache-intercept-ip intercept.v1 '{
    "protocols": ["tcp"],
    "addresses": ["10.100.99.99"],
    "portRanges": [{"low": 80, "high": 80}]
}'

ziti edge create config apache-host.v1 host.v1 '{
    "protocol": "tcp",
    "address": "127.0.0.1",
    "port": 80
}'

ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1 # Did not add an attribute here as you suggested.

ziti edge create service-policy apache-dial-policy Dial \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-private"

ziti edge create service-policy apache-bind-policy Bind \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-public"

I used the service name directly in the policy as you suggested:

ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
      --edge-router-roles '#router-services' \
      --service-roles '@apache-service'

Results:

Screen Session Log on EC2-Public:

About to run tunnel service... ziti-edge-tunnel
(2236)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(2236)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2025-01-03T16:13:26.912)
(2236)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(2236)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(2236)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(2236)[        0.024]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(2236)[        0.025]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-public.json] is not loaded yet or already removed.
(2236)[        0.025]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-public.json]
(2236)[        0.025]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-public.json]
(2236)[        0.025]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-public.json] loaded
(2236)[        0.025]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(2236)[        0.025]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(2236)[        0.025]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(2236)[        0.025]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(2236)[        0.084]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(2236)[        0.098]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-public] connected to controller
(2236)[        0.098]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-public.json] context event : status is OK
(2236)[        0.130]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-public) new channel for ztx[0] identity[EC2-public]
(2236)[        0.130]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-public] added edge router router-public@ziti-router-public.example.com
(2236)[        0.130]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(2236)[        0.237]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:953 on_service() hosting server_address[tcp:127.0.0.1:80] service[apache-service]
(2236)[        0.237]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:4Qrx582kuj8EMabc4Y9sSl ===============
(2236)[        0.237]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
(2236)[        0.243]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[        0.268]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
(2236)[        0.271]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(2236)[        0.271]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-public] router router-public connected
(2236)[        0.271]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(2236)[        0.956]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[        1.093]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(2236)[        2.844]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[        9.507]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE

Screen Session Log on EC2-Private:

About to run tunnel service... ziti-edge-tunnel
(1582)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(1582)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2025-01-03T16:10:05.779)
(1582)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(1582)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(1582)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(1582)[        0.028]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-private.json] is not loaded yet or already removed.
(1582)[        0.028]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(1582)[        0.028]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-private.json]
(1582)[        0.028]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-private.json]
(1582)[        0.028]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-private.json] loaded
(1582)[        0.030]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(1582)[        0.030]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(1582)[        0.030]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(1582)[        0.030]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(1582)[        0.208]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(1582)[        0.228]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:798 on_event() ztx[EC2-private.json] API Event with controller address : (null)
(1582)[        0.231]   ERROR tunnel-cbs:ziti_tunnel_ctrl.c:1571 update_config_done() updated config file with new URL
(1582)[        0.232]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-private] connected to controller
(1582)[        0.232]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-private.json] context event : status is OK
(1582)[        0.267]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-private) new channel for ztx[0] identity[EC2-private]
(1582)[        0.267]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-private] added edge router router-private@ziti-router-private.example.com
(1582)[        0.267]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(1582)[        0.482]    INFO tunnel-cbs:ziti_tunnel_cbs.c:414 new_ziti_intercept() creating intercept for service[apache-service] with intercept.v1 = { "addresses": [ "10.100.99.99" ], "portRanges": [ { "high": 80, "low": 80 } ], "protocols": [ "tcp" ] }
(1582)[        0.482]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:938 on_service() starting intercepting for service[apache-service]
(1582)[        0.482]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:4Qrx582kuj8EMabc4Y9sSl ===============
(1582)[        0.482]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 2 route updates
(1582)[        0.486]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[2]: 0/OK
(1582)[        0.525]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(1582)[        0.525]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-private] router router-private connected
(1582)[        0.525]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(1582)[        1.228]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected

Here are my current edge router and service configurations:

$ ziti edge list edge-routers
# Outputs details of all edge routers (router-public, router-private, router-services)
╭────────────┬─────────────────┬────────┬───────────────┬──────┬─────────────────╮
│ ID         │ NAME            │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES      │
├────────────┼─────────────────┼────────┼───────────────┼──────┼─────────────────┤
│ 5TL8AjQpd9 │ router-private  │ true   │ true          │    0 │ router-private  │
│ AQy8Ajppd9 │ router-services │ true   │ true          │    0 │ router-services │
│ lgB8NUppw9 │ router-public   │ true   │ true          │    0 │ router-public   │
╰────────────┴─────────────────┴────────┴───────────────┴──────┴─────────────────╯
results: 1-3 of 3

$ ziti edge list service service-edge-router-policies apache-service
# Shows no issues with the service edge router policy.
╭────────────────────────┬─────────────────────────────────────────┬─────────────────┬───────────────────╮
│ ID                     │ NAME                                    │ SERVICE ROLES   │ EDGE ROUTER ROLES │
├────────────────────────┼─────────────────────────────────────────┼─────────────────┼───────────────────┤
│ 30NF1lLnUm0yzylPeSLiLi │ all-services-on-router-services-policy2 │ @apache-service │ #router-services  │
╰────────────────────────┴─────────────────────────────────────────┴─────────────────┴───────────────────╯
results: 1-1 of 1

Could you please guide me on what might be missing or misconfigured in the current setup?

scareything · January 3, 2025, 5:26pm

Your identities are only allowed to access router-private and router-public respectively. These routers are not permitted to carry apache-service. The only router that can handle apache-service is the new router-services router, but I don't see any edge router policies that allow your identities to use the router-services router.

It looks like you either need to allow router-private and router-public to handle apache-service with service-edge-router policies, or allow your identities to use the router-services router with edge router policies.

am3y · January 3, 2025, 5:41pm

Ohh, I thought setting up a separate router for services (router-services) and dedicated routers for the EC2 instances (router-private and router-public) would be a better approach to segregate traffic.

But based on your feedback, it seems my way of doing it might not align with how OpenZiti operates, right?

scareything · January 3, 2025, 6:12pm

A router needs to be accessible by the identity and the service to be a viable path.

The only thing that seems odd to me is the way you're creating identities, routers, and (previously) services with a role attribute that matches the name of the entity:

ziti edge create identity device EC2-private --role-attributes "EC2-private" -o EC2-private.jwt

I may be missing something that's important to you, but in my opinion a role-attribute that is only going to be assigned to a single entity is unnecessary, because you can just use the entity name with @ instead. To me, role attribues are useful when you want to "group" more than one entity together, such as '#public-routers' (plural), or #boston-users where you add the attribute to more than one router/identity etc.

am3y · January 3, 2025, 6:47pm

Thanks for your insights @scareything . Let me clarify the structure I'm working towards, and I'd appreciate your suggestions if my approach needs improvement.

Current Structure

I'm currently exploring OpenZiti and planning to set it up across multiple EC2 instances divided into zones. Here's how I'm organizing it:

Example: Zone-A

EC2 Instances:
All EC2 instances in Zone-A are within the same VPC as the EKS cluster. For these, I create identities like:

ziti edge create identity device EC2-private-zoneA-app1 --role-attributes "EC2-private" -o EC2-private-zoneA-app1.jwt
ziti edge create identity device EC2-private-zoneA-app2 --role-attributes "EC2-private" -o EC2-private-zoneA-app2.jwt
ziti edge create identity device EC2-private-zoneA-app3 --role-attributes "EC2-private" -o EC2-private-zoneA-app3.jwt

Routers:
I set up routers with internal-facing load balancers (to ensure traffic between the EC2 instances and the routers stays within the VPC). For example:

ziti edge create edge-router router-private-zone-a-1 --role-attributes "router-private-zone-a" --tunneler-enabled --jwt-output-file router-private-zone-a-1.jwt
ziti edge create edge-router router-private-zone-a-2 --role-attributes "router-private-zone-a" --tunneler-enabled --jwt-output-file router-private-zone-a-2.jwt

Both routers share the router-private-zone-a attribute.

Router Policies:
Any EC2 with the #EC2-private attribute can connect to any router with the #router-private attribute:

ziti edge create edge-router-policy router-private-router-policy \
--edge-router-roles "#router-private" \
--identity-roles "#EC2-private" \
--semantic "AllOf"

This setup ensures scalability and flexibility since any new EC2 instance or router in Zone-A can just have the appropriate attributes assigned to integrate into the system seamlessly.

About Services

I was thinking of replicating a similar approach for services by having a dedicated set of routers (e.g., router-services) specifically for handling service traffic.

Question

IDK if my approach is right or wrong, but I thought it made sense for scalability and flexibility.

Do you think this is a good way to structure the setup, or would you recommend a different approach? I'm still exploring OpenZiti and would value your suggestions to align with best practices.

am3y · January 3, 2025, 6:49pm

IDK if I'm right or wrong but I was thinking to setup stuff like this

am3y · January 5, 2025, 8:30pm

Just following up to see if anyone has insights—thanks in advance!

TheLumberjack · January 6, 2025, 2:41pm

Hi @am3y,

OpenZiti allows you to have a lot of flexibility. Part ot my problem are generic terms like "load balancer" have overloaded meanings. It's generally easier for us to say, "yes that makes sense" from a generic diagram similar to the one you provided.

If I were to look at your diagram, I would change it because as it appears, it doesn't quite make sense to me. The router-public appears to be within private address space, meaning it won't be usable from outside of AWS. Maybe that's what you want, but I don't know for sure. It would be atypical, but maybe that's what you want?

A more typical style of deployment imo would have the controller and the public-router deployed on the right-hand side of your diagram in the 'public' vpc. Then, I would keep one or more routers deployed within the private vpc, and these routers would connect out to the public router(s), forming the secure OpenZiti overlay mesh and providing the (generally) desired reachability from anywhere on the open internet.

That make sense?

am3y · January 6, 2025, 2:46pm

Hii @TheLumberjack
I guess I have already cleared this in the start. Ignore the public & private thing..

TheLumberjack · January 6, 2025, 2:49pm

Then it sounds like it'll be just fine, yes.

am3y · January 6, 2025, 3:26pm

Yeah but idk why it's not working

TheLumberjack · January 6, 2025, 3:30pm

I don't understand. As of Jan 2 you reported:

And in the comment I replied to, you had questions, but none of those questions indicated "something wasn't working".

If something isn't working, can you reply with the error and "what" isn't working? I scanned back through and I didn't see it mentioned, but maybe I just missed it.

am3y · January 6, 2025, 3:42pm

I thought you have gone through this

TheLumberjack · January 6, 2025, 8:07pm

Is that the current issue then? Assuming that's the case, I would strongly suggest you revise your service-edge-router-policy (SERP) for the time being and use an #all/#all policy. I think the SERP is misconfigured. The service-edge-router-policy object is always hard for people to understand at first and is easy to misconfigure.

A service edge router policy (SERP) indicates the routers a service must enter or exit the openziti overlay from. Looking at your SERP, it looks to me that you have allowed the service to onboard to the overlay on router-services only. However, the edge-router policies you created only gives identities access to the two routers (router-private/router-public) which are NOT router-services. I think that's the problem.

In general, it's always easier to start open with OpenZiti's policies and refine them to be more granular after you get used to using OpenZiti. I really do suggest starting out with a SERP that's #all/#all and remove it from the equation for simplicity's sake. However, I think if you add the other two routers (and remove router-services to your policy) it should work.

Topic		Replies	Views
Ziti Tunneler Configuration: Resolving Identical ziti0 IPs and Enabling Cross-EC2 Communication Ziti Overlay	4	36	December 28, 2024
Need Guidance on Setting Up Multiple OpenZiti Routers and Identity Devices in EKS	4	37	August 9, 2024
EKS Cluster Router :"failed to dial fabric Ziti Overlay	64	118	July 25, 2024
AWS Self hosted K8s/DB scenarios General Questions	6	30	November 1, 2024
Intermittent Connectivity Issues: ziti-edge-tunnel Support	4	27	November 21, 2024