Ziti Tunneler Configuration: Resolving Identical ziti0 IPs and Enabling Cross-EC2 Communication

Ziti Overlay
1

Disclaimer:

Due to a non-disclosure agreement (NDA), the details in this post have been modified and are based on a hypothetical example for illustrative purposes. The core issue, however, remains representative of the actual challenge.

Scenario:

  • Two VPCs are configured: VPC-public and VPC-private.
  • Two EC2 instances are present:
    • EC2-public in VPC-public.
    • EC2-private in VPC-private.
  • An EKS cluster is set up in VPC-private.

The following OpenZiti components have been deployed:

Ziti Controller

The controller was deployed with the following values.yml configuration:

clientApi:
  advertisedHost: ziti-controller.example.com
  service:
    enabled: true
    type: ClusterIP

  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      external-dns.alpha.kubernetes.io/hostname: "ziti-controller.example.com"
      service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # Ensures the LB is public


ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP




highAvailability:
  # -- Ziti controller HA mode
  mode: standalone
  # -- Ziti controller HA swarm replicas
  replicas: 1


persistence:
  enabled: true
  storageClass: "ebs-sc"
  accessMode: ReadWriteOnce
  size: 3Gi


cert-manager:
  enabled: true
  enableCertificateOwnerRef: true
  installCRDs: false

trust-manager:
  enabled: true
  app:
    trust:
      namespace: "ziti-controller"
  crds:
    enabled: false

ingress-nginx:
  enabled: true
  controller:
    extraArgs:
      enable-ssl-passthrough: "true"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"

Ziti Routers

Two routers were set up:

  1. Router-Private:

     ctrl:
   endpoint: ziti-controller.example.com:443
   advertisedHost: ziti-router-private.example.com

 # Edge configuration for external identities
 edge:
   advertisedHost: ziti-router-private.example.com
   advertisedPort: 443
   service:
     type: LoadBalancer  
     annotations:
       external-dns.alpha.kubernetes.io/hostname: ziti-router-private.example.com
       service.beta.kubernetes.io/aws-load-balancer-internal: "true" # loadbalancer here is private
   ingress:
     enabled: false

 # Link listeners for router-to-router communication (internal)
 linkListeners:
   transport:
     advertisedHost: ziti-router-transport-private.example.com
     advertisedPort: 443
     service:
       enabled: true
       type: ClusterIP  # All routers are internal; no external exposure
     ingress:
       enabled: false  # Not needed as routers are internal

 # Persistence for router data
 persistence:
   enabled: true
   accessMode: ReadWriteOnce
   size: 1Gi
   storageClass: ebs-sc

  2. Router-Public:

     ctrl:
   endpoint: ziti-controller.example.com:443
   advertisedHost: ziti-router-public.example.com

 # Edge configuration for external identities
 edge:
   advertisedHost: ziti-router-public.example.com
   advertisedPort: 443
   service:
     type: LoadBalancer  
     annotations:
       external-dns.alpha.kubernetes.io/hostname: ziti-router-public.example.com
       service.beta.kubernetes.io/aws-load-balancer-internal: "false" # Loadbalancer here is public
   ingress:
     enabled: false

 # Link listeners for router-to-router communication (internal)
 linkListeners:
   transport:
     advertisedHost: ziti-router-transport-public.example.com
     advertisedPort: 443
     service:
       enabled: true
       type: ClusterIP  # All routers are internal; no external exposure
     ingress:
       enabled: false  # Not needed as routers are internal

 # Persistence for router data
 persistence:
   enabled: true
   accessMode: ReadWriteOnce
   size: 1Gi
   storageClass: ebs-sc

The routers were installed using Helm with enrollment JWTs:

helm install ziti-router-private-release \
  --namespace ziti-router --create-namespace \
  openziti/ziti-router \
  --set-file enrollmentJwt=router-private.jwt \
  --values router-values-private.yml

helm install ziti-router-public-release \
  --namespace ziti-router --create-namespace \
  openziti/ziti-router \
  --set-file enrollmentJwt=router-public.jwt \
  --values router-values-public.yml

Ziti Edge Tunnel Configuration

  • The identity for EC2-public was created, enrolled, and the Ziti tunnel was started:

    ziti edge create identity device EC2-public --role-attributes "EC2-public" -o EC2-public.jwt
ziti edge enroll EC2-public.jwt -o EC2-public.json
ziti-tunnel ziti-edge-tunnel run -i EC2-public.json

  • The identity for EC2-private was created, enrolled, and the Ziti tunnel was started:

    ziti edge create identity device EC2-private --role-attributes "EC2-private" -o EC2-private.jwt
ziti edge enroll EC2-private.jwt -o EC2-private.json
ziti-tunnel ziti-edge-tunnel run -i EC2-private.json

  • Edge router policies were created to bind each EC2 instance to its respective router:

    ziti edge create edge-router-policy router-private-router-policy \
  --edge-router-roles "#router-private" \
  --identity-roles "#EC2-private" \
  --semantic "AllOf"

ziti edge create edge-router-policy router-public-router-policy \
  --edge-router-roles "#router-public" \
  --identity-roles "#EC2-public" \
  --semantic "AllOf"

Observations:

  1. Both EC2 instances (EC2-public and EC2-private) successfully connect to their respective routers, as shown in the ziti-edge-tunnel logs.
  2. The ziti0 interface is active on both EC2 instances but is assigned the same IP (100.64.0.1).
  3. The following tests were performed:
    • Attempting to curl the public-facing application on EC2-public using its Ziti tunnel IP (100.64.0.1) failed:
      curl -I --interface ziti0 http://100.64.0.1/index.html
    • Attempting to curl the application using the private network IP of EC2-public also failed:
      curl -I --interface ziti0 http://10.10.10.29/index.html

Logs and Outputs

  • ziti-edge-tunnel logs from EC2-public:

    About to run tunnel service... ziti-edge-tunnel
(2392157)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(2392157)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2024-12-20T15:38:21.919)
(2392157)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.9)
(2392157)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(2392157)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:894 make_socket_path() effective group set to 'ziti' (gid=995)
(2392157)[        0.054]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(2392157)[        0.054]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-public] is not loaded yet or already removed.
(2392157)[        0.054]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-public]
(2392157)[        0.054]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-public]
(2392157)[        0.054]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:402 load_id_cb() identity[EC2-public] loaded
(2392157)[        0.065]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(2392157)[        0.065]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(2392157)[        0.065]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(2392157)[        0.065]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(2392157)[        0.482]    WARN ziti-edge-tunnel:resolvers.c:399 try_libsystemd_resolver() libsystemd resolver unsuccessful. Falling back to legacy resolvers
(2392157)[        0.847]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(2392157)[        1.369]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-public] connected to controller
(2392157)[        1.369]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:440 on_event() ztx[EC2-public] context event : status is OK
(2392157)[        6.685]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(2392157)[        7.199]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-public) new channel for ztx[0] identity[EC2-public]
(2392157)[        7.199]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-public] added edge router router-public@ziti-router-public.example.com
(2392157)[        7.199]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(2392157)[       13.353]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
(2392157)[       13.370]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
(2392157)[       13.426]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(2392157)[       13.426]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-public] router router-public connected

  • ziti-edge-tunnel logs from EC2-private:

    About to run tunnel service... ziti-edge-tunnel
(2758075)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(2758075)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2024-12-20T15:37:58.608)
(2758075)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.9)
(2758075)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(2758075)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:894 make_socket_path() effective group set to 'ziti' (gid=989)
(2758075)[        0.007]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-private] is not loaded yet or already removed.
(2758075)[        0.007]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(2758075)[        0.007]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-private]
(2758075)[        0.007]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-private]
(2758075)[        0.007]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:402 load_id_cb() identity[EC2-private] loaded
(2758075)[        0.007]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(2758075)[        0.007]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(2758075)[        0.007]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(2758075)[        0.007]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(2758075)[        0.049]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(2758075)[        0.063]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-private] connected to controller
(2758075)[        0.063]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:440 on_event() ztx[EC2-private] context event : status is OK
(2758075)[        0.134]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-private) new channel for ztx[0] identity[EC2-private]
(2758075)[        0.134]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-private] added edge router router-private@ziti-router-private.example.com
(2758075)[        0.134]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(2758075)[        0.141]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
(2758075)[        0.143]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
(2758075)[        0.165]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(2758075)[        0.165]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-private] router router-private connected
(2758075)[        0.165]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(2758075)[        1.058]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected

  • ifconfig output from EC2-public:

    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.10.29  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::ba27:ebff:fe0c:a935  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:0c:a9:35  txqueuelen 1000  (Ethernet)
        RX packets 440588377  bytes 522850420062 (486.9 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 215353337  bytes 17076616432 (15.9 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 36013881  bytes 5975687878 (5.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36013881  bytes 5975687878 (5.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ziti0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 100.64.0.1  netmask 255.255.255.255  destination 100.64.0.1
        inet6 fe80::89c2:7484:c7b:9637  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 2919  bytes 297279 (290.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2937  bytes 273446 (267.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

  • ifconfig output from EC2-private:

    ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 192.168.14.181  netmask 255.255.240.0  broadcast 192.168.15.255
        inet6 fe80::e0:10ff:fe11:5eb1  prefixlen 64  scopeid 0x20<link>
        ether 02:e0:10:11:5e:b1  txqueuelen 1000  (Ethernet)
        RX packets 7418657  bytes 1865728819 (1.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7933658  bytes 3494499519 (3.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 192.168.65.56  netmask 255.255.224.0  broadcast 192.168.95.255
        inet6 fe80::8e:7bff:fec9:711f  prefixlen 64  scopeid 0x20<link>
        ether 02:8e:7b:c9:71:1f  txqueuelen 1000  (Ethernet)
        RX packets 1659663  bytes 1590998141 (1.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1315295  bytes 111982452 (106.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 84  bytes 10614 (10.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 84  bytes 10614 (10.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ziti0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 100.64.0.1  netmask 255.255.255.255  destination 100.64.0.1
        inet6 fe80::eb3a:226:1135:c53a  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 15655  bytes 1315020 (1.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15669  bytes 1315692 (1.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Requirements:

  1. Enable communication from EC2-private to EC2-public over the Ziti overlay to access the application (Apache in this case) running on EC2-public.
  2. Resolve the issue of both ziti0 interfaces having the same IP (100.64.0.1), which prevents proper routing and makes direct communication impossible.
  3. Provide a step-by-step guide for configuring the necessary Ziti services and policies to facilitate this setup.

Questions:

  • What specific Ziti service and policy configurations are required to enable this communication?
  • How can the issue of identical IPs (100.64.0.1) for the ziti0 interfaces be addressed effectively?
  • Is there a recommended approach to debug and ensure successful routing for the traffic in this scenario?

Any guidance or insights to resolve this issue would be greatly appreciated. Thank you!

2

Hi @am3y,

This here makes me think there's a fundamental gap happening. It makes want to ask you to restate your goals because for me, as someone who understands OpenZiti, this information just isn't useful because I expect that curl to fail... :slight_smile: From an IP-perspective, It's understandable that it'd be confusing, but it doesn't surprise me that the curl doesn't respond...

To your requirements:

This sounds like a common design pattern. A diagram of your desired topology would be the most helpful thing for me, if you have one?

I thank you for the very detailed response, but if we take it back to the very basic desire, that will be more helpful to me I think.

OpenZiti has some concepts that don't really apply to classic, IP-based routing so I think describing the situation first is step one. Then, establishing the overlay network is step two. Finally, enabling your desired outcome is step 3 to me.

Can we start with a simple description of the problem you're trying to accomplish? Or - can you confirm my perspecitive is correct? You're looking to have:

  • OpenZiti controller deployed into public IP space
  • OpenZiti router deployed into public IP space
  • OpenZiti router deployed into private IP space
  • Access an Apache Server in that private IP space

Is that what you're doing? I'd also like to know where the client is that will be accessing the apache server.

Cheers

Something like this maybe that illustrates what you're doing is useful (this is from a different discourse post but it's similar)

image
image3280×1364 405 KB

3

Hi @TheLumberjack

Thank you for your thoughtful reply and for clarifying OpenZiti’s design approach! I now see that my example relied too heavily on traditional IP-based concepts, which might not fully align with OpenZiti's overlay model.

To address this:

  1. I will revise the example to better reflect OpenZiti's architecture, focusing on identity-based routing and service definitions.
  2. I will also create a network diagram to illustrate the topology and desired communication flows, as you suggested.

Would you recommend that I update this thread with the revised details and diagram or create a new post? I’m happy to proceed in whichever way works best for the community.

Thank you again for your guidance and patience!

Cheers,
Amey

4

Just a real rough sketch/diagram would help out, it doesn't need any polish and I'd say just post it right back here on this thread, no need to make a new one yet. :slight_smile:

5

@TheLumberjack
Thank you for your earlier response and guidance. I’ve been troubleshooting further and exploring the concepts in more depth, which has clarified a few things for me—like why an identical ziti0 IP is assigned to every identity.

However, the scope of my issue has evolved, and I felt editing this post wouldn’t be appropriate, as the title and topic no longer align with the current problem. To avoid confusion and ensure the discussion remains focused, I’ve created a new post detailing my updated issue and findings.

You can find the new post here: [New Post].

I’d really appreciate your insights on the updated scenario if you have a chance to take a look. Thank you again for your support!