# Troubleshooting OpenZiti Tunnel: Edge Router and Service Configuration Issues

**URL:** https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683
**Category:** Uncategorized
**Created:** [December 28, 2024, 12:09am UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683 "2024-12-28T00:09:43Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [December 28, 2024, 12:09am UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/1 "2024-12-28T00:09:43Z")

</div>

### **Disclaimer:**

Due to a non-disclosure agreement (NDA), the details in this post have been modified and are based on a hypothetical example for illustrative purposes. The core issue, however, remains representative of the actual challenge.

* * *

### **Scenario:**

- Two VPCs are configured: `VPC-public` and `VPC-private`. Note: These public, private are just names for our convinence.
- Two EC2 instances are present:
  - `EC2-public` in `VPC-public`.
  - `EC2-private` in `VPC-private`.

- An EKS cluster is set up in `VPC-private`.  
I have also posted a simple diagram to understand the architecture

The following OpenZiti components have been deployed:

#### **Ziti Controller**

The controller was deployed with the following `values.yml` configuration:

```yaml
clientApi:
  advertisedHost: ziti-controller.example.com
  service:
    enabled: true
    type: ClusterIP

  ingress:
    enabled: true
    ingressClassName: "nginx"
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      external-dns.alpha.kubernetes.io/hostname: "ziti-controller.example.com"
      service.beta.kubernetes.io/aws-load-balancer-internal: "false" # Ensures the LB is public

ctrlPlane:
  containerPort: "{{ .Values.clientApi.containerPort }}"
  advertisedHost: "{{ .Values.clientApi.advertisedHost }}"
  advertisedPort: "{{ .Values.clientApi.advertisedPort }}"
  service:
    enabled: true
    type: ClusterIP

highAvailability:
  # -- Ziti controller HA mode
  mode: standalone
  # -- Ziti controller HA swarm replicas
  replicas: 1

persistence:
  enabled: true
  storageClass: "ebs-sc"
  accessMode: ReadWriteOnce
  size: 3Gi

cert-manager:
  enabled: true
  enableCertificateOwnerRef: true
  installCRDs: false

trust-manager:
  enabled: true
  app:
    trust:
      namespace: "ziti-controller"
  crds:
    enabled: false

ingress-nginx:
  enabled: true
  controller:
    extraArgs:
      enable-ssl-passthrough: "true"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"

```

#### **Ziti Routers**

Two routers were set up:

1. **Router-Private** :

2. **Router-Public** :

The routers were installed using Helm with enrollment JWTs:

```bash
helm install ziti-router-private-release \
  --namespace ziti-router --create-namespace \
  openziti/ziti-router \
  --set-file enrollmentJwt=router-private.jwt \
  --values router-values-private.yml

helm install ziti-router-public-release \
  --namespace ziti-router --create-namespace \
  openziti/ziti-router \
  --set-file enrollmentJwt=router-public.jwt \
  --values router-values-public.yml

```

#### **Ziti Edge Tunnel Configuration**

- The identity for `EC2-public` was created, enrolled, and the Ziti tunnel was started:

- The identity for `EC2-private` was created, enrolled, and the Ziti tunnel was started:

- Edge router policies were created to bind each EC2 instance to its respective router:

* * *

### **Our Task**

So in `EC2-public` we have a application which is accessed on TCP:80. We want to access that application from `EC2-private`. For testing we have installed apache2 where we can just run the curl command to test.

So to achieve this what I ran the below commands:

#### Intercept Configuration

```bash
ziti edge create config apache-intercept-ip intercept.v1 '{
   "protocols": ["tcp"],
   "addresses": ["10.100.99.99"],
   "portRanges": [{"low": 80, "high": 80}]
}'

```

#### Host Configuration

```bash
ziti edge create config apache-host.v1 host.v1 '{
   "protocol": "tcp",
   "address": "127.0.0.1",
   "port": 80
}'

```

#### Create the Service

```bash
ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1

```

#### Create Service Policies

- **Dial Policy** : Allow `EC2-private` to access the service:

- **Bind Policy** : Allow `EC2-public` to host the service:

#### Connecting the tunnels:

```bash
ziti-edge-tunnel run -i EC2-public.json
ziti-edge-tunnel run -i EC2-private.json

```

* * *

Now both the tunnels get connected, technically I have to run curl [http://10.100.99.99](http://10.100.99.99) to test the connection here. But it is not connecting.  
After checking the tunnels logs I have found that there is some issue in the `EC2-public`

#### **Logs and Outputs**

- `ziti-edge-tunnel` logs from **EC2-public** :

- `ziti-edge-tunnel` logs from **EC2-private** :

* * *

Any guidance or insights to resolve this issue would be greatly appreciated. Thank you!

 ![openziti2.drawio](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/c/c445cfd635bae4cf7af5a4a6bc02191d144928b5.png)

---

<div class="post-metadata">

### Author: ![scareything](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/scareything/32/106_2.png) [@scareything](https://openziti.discourse.group/u/scareything)
#### Post date: [December 29, 2024, 10:34pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/2 "2024-12-29T22:34:19Z")

</div>

> [@am3y](#):
>
> ```auto
> (7088)[2.130] WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
> 
> ```

This looks like the identity is not allowed to use any edge routers. Have you created any edge router policies?

You can get a quick summary of your overall policy setup with `ziti edge policy-advisor`. Does it show any issues?

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [December 30, 2024, 12:09am UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/3 "2024-12-30T00:09:19Z")

</div>

Thanks for your reply @scareything

Here is the output of that command

**$ ziti edge policy-advisor services**

```auto
Policy General Guidelines
  In order for an identity to dial or bind a service, the following must be true:
    - The identity must have access to the service via a service policy of the correct type (dial or bind)
    - The identity must have access to at least one on-line edge router via an edge router policy
    - The service must have access to at least one on-line edge router via a service edge router policy
    - There must be at least one on-line edge router that both the identity and service have access to.

Policy Advisor Output Guide:
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR. 
  ID = identity name
  ID ROUTERS = number of routers accessible to the identity via edge router policies.
    - See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
  SVC = service name
  SVC ROUTERS = number of routers accessible to the service via service edge router policies.
    - See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
  ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
  COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
  DIAL_OK = indicates if the identity has permission to dial the service.
    - See service polices for a service : ziti edge controller list service service-policies <service>
    - See service polices for an identity: ziti edge controller list identity service-policies <identity>
  BIND_OK = indicates if the identity has permission to bind the service.
  ERROR_LIST = if the status is ERROR, error details will be listed on the following lines

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
ERROR: EC2-private (1) -> apache-service (0) Common Routers: (0/0) Dial: Y Bind: N 
  - Service has no edge routers assigned. Adjust service edge router policies.

ERROR: EC2-public (1) -> apache-service (0) Common Routers: (0/0) Dial: N Bind: Y 
  - Service has no edge routers assigned. Adjust service edge router policies.

```

**$ ziti edge policy-advisor identities**

```auto
Policy General Guidelines
  In order for an identity to dial or bind a service, the following must be true:
    - The identity must have access to the service via a service policy of the correct type (dial or bind)
    - The identity must have access to at least one on-line edge router via an edge router policy
    - The service must have access to at least one on-line edge router via a service edge router policy
    - There must be at least one on-line edge router that both the identity and service have access to.

Policy Advisor Output Guide:
  STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR. 
  ID = identity name
  ID ROUTERS = number of routers accessible to the identity via edge router policies.
    - See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
  SVC = service name
  SVC ROUTERS = number of routers accessible to the service via service edge router policies.
    - See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
  ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
  COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
  DIAL_OK = indicates if the identity has permission to dial the service.
    - See service polices for a service : ziti edge controller list service service-policies <service>
    - See service polices for an identity: ziti edge controller list identity service-policies <identity>
  BIND_OK = indicates if the identity has permission to bind the service.
  ERROR_LIST = if the status is ERROR, error details will be listed on the following lines

Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
ERROR: EC2-private (1) -> apache-service (0) Common Routers: (0/0) Dial: Y Bind: N 
  - Service has no edge routers assigned. Adjust service edge router policies.

ERROR: EC2-public (1) -> apache-service (0) Common Routers: (0/0) Dial: N Bind: Y 
  - Service has no edge routers assigned. Adjust service edge router policies.

ERROR: Default Admin 
  - Identity does not have access to any services. Adjust service policies.

ERROR: router-private 
  - Identity does not have access to any services. Adjust service policies.

ERROR: router-public 
  - Identity does not have access to any services. Adjust service policies.

```

---

<div class="post-metadata">

### Author: ![scareything](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/scareything/32/106_2.png) [@scareything](https://openziti.discourse.group/u/scareything)
#### Post date: [December 30, 2024, 11:54am UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/4 "2024-12-30T11:54:47Z")

</div>

Similar to how you can restrict access to services based on identities or attributes, OpenZiti also allows you to limit which services an edge router may handle connections for, and which identities may use the edge router. Edge router access is controlled with `service-edge-router-policy` and `edge-router-policy`. A special roll `#all` may be used here if you want to allow edge routers to be used for all services and/or by all identities:

```bash
ziti edge create service-edge-router-policy all-services-on-all-routers --edge-router-roles '#all' --service-roles '#all'

```

```bash
ziti edge create edge-router-policy all-identities-on-all-routers --edge-router-roles '#all' --identity-roles '#all'

```

Of course you can use more specific identity/service/router roles as your situation requires.

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 2, 2025, 9:13pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/5 "2025-01-02T21:13:41Z")

</div>

@scareything Thank you so much for your detailed response and guidance! I appreciate the time and effort you've put into helping me troubleshoot this issue. 🙏

I followed your advice and tested with the following commands:

```auto
ziti edge create service-edge-router-policy all-services-on-all-routers \
      --edge-router-roles '#all' \
      --service-roles '#all'

ziti edge create edge-router-policy all-identities-on-all-routers \
      --edge-router-roles '#all' \
      --identity-roles '#all'

```

With these broad policies, I was able to successfully test the service using:

```auto
curl http://10.100.99.99/

```

* * *

### Setting Up Specific Routers

Since I wanted to restrict specific routers to handle traffic based on their usage, I created a new router named `router-services` with the following configuration:

```auto
ctrl:
  endpoint: ziti-controller.example.com:443
  advertisedHost: ziti-router-services.ziti-router.svc.cluster.local

# Edge configuration for external identities
edge:
  advertisedHost: ziti-router-services.ziti-router.svc.cluster.local
  advertisedPort: 443
  service:
    type: ClusterIP
  ingress:
    enabled: false

# Link listeners for router-to-router communication (internal)
linkListeners:
  transport:
    advertisedHost: ziti-router-transport-services.ziti-router.svc.cluster.local
    advertisedPort: 443
    service:
      enabled: true
      type: ClusterIP
    ingress:
      enabled: false

```

Additionally, I applied the following **edge-router policies** to restrict EC2 instances to their respective routers:

```auto
ziti edge create edge-router-policy router-private-router-policy \
--edge-router-roles "#router-private" \
--identity-roles "#EC2-private" \
--semantic "AllOf"

ziti edge create edge-router-policy router-public-router-policy \
--edge-router-roles "#router-public" \
--identity-roles "#EC2-public" \
--semantic "AllOf"

```

I set up the service and policies as follows:

```auto
ziti edge create config apache-intercept-ip intercept.v1 '{
    "protocols": ["tcp"],
    "addresses": ["10.100.99.99"],
    "portRanges": [{"low": 80, "high": 80}]
}'

ziti edge create config apache-host.v1 host.v1 '{
    "protocol": "tcp",
    "address": "127.0.0.1",
    "port": 80
}'

ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1

ziti edge create service-policy apache-dial-policy Dial \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-private"

ziti edge create service-policy apache-bind-policy Bind \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-public"

```

* * *

### Test Cases and Issues

#### **CASE-1: Specific Router Services Policy**

I used this command to configure the service to use `router-services`:

```auto
ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
      --edge-router-roles '#router-services' \
      --service-roles '#apache-service'

```

On the `EC2-public` screen session, I observed the following repeated error:

```auto
WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE

```

In the `EC2-private` screen session, no errors were logged.

* * *

#### **CASE-2: Broad Identity Policy**

I reverted to the following policy:

```auto
ziti edge create edge-router-policy all-identities-on-all-routers \
      --edge-router-roles '#all' \
      --identity-roles '#all'

```

This caused the following error in the `EC2-private` screen session:

```auto
ERROR ziti-sdk:connect.c:1071 connect_reply_cb() conn[0.0/xxeoGuM2/Connecting](apache-service) failed to connect, reason=can't route from SrmOpTMM28 -> RhfQp6BBU8

```

* * *

### Current Setup

Here is the output of the edge routers:

```auto
ziti edge list edge-routers
╭────────────┬─────────────────┬────────┬───────────────┬──────┬─────────────────╮
│ ID │ NAME │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├────────────┼─────────────────┼────────┼───────────────┼──────┼─────────────────┤
│ 6ofQaTMMU8 │ router-services │ true │ true │ 0 │ router-services │
│ RhfQp6BBU8 │ router-public │ true │ true │ 0 │ router-public │
│ SrmOpTMM28 │ router-private │ true │ true │ 0 │ router-private │
╰────────────┴─────────────────┴────────┴───────────────┴──────┴─────────────────╯

```

* * *

### Questions

1. **Router-Services Configuration** :

2. **Service Policies** :

3. **Routing Errors** :

Could you help identify what’s causing these issues and suggest any changes? Thank you in advance! 🚀

---

<div class="post-metadata">

### Author: ![scareything](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/scareything/32/106_2.png) [@scareything](https://openziti.discourse.group/u/scareything)
#### Post date: [January 3, 2025, 3:13pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/6 "2025-01-03T15:13:57Z")

</div>

> [@am3y](#):
>
> I set up the service and policies as follows:
> 
> ```auto
> ...
> ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1
> 
> ```

The _name_ of your service is `apache-service`, but you are specifying an attribute (with '#') when you create the service edge router policy:

> [@am3y](#):
>
> ```auto
> ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
> --edge-router-roles '#router-services' \
> --service-roles '#apache-service'
> 
> ```

And from what I can see you didn't assign any attributes to that service. So if I'm right about that then you could either add the attribute to your service:

```auto
ziti edge update service apache-service --role-attributes apache-service

```

You may have a reason for doing this, but it seems redundant to me at first glance. I'd suggest not creating attributes that mirror the names of your entities, and using the entity names directly (by prefixing with '@') when the intent is to specify a specific identity/service/router. For example:

```auto
ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
      --edge-router-roles '#router-services' \
      --service-roles '@apache-service'

```

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 3, 2025, 4:22pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/7 "2025-01-03T16:22:55Z")

</div>

Thanks for your speedy reply @scareything . I tried the following steps as you suggested:

```bash
ziti edge create edge-router-policy router-private-router-policy \
--edge-router-roles "#router-private" \
--identity-roles "#EC2-private" \
--semantic "AllOf"

ziti edge create edge-router-policy router-public-router-policy \
--edge-router-roles "#router-public" \
--identity-roles "#EC2-public" \
--semantic "AllOf"

ziti edge create config apache-intercept-ip intercept.v1 '{
    "protocols": ["tcp"],
    "addresses": ["10.100.99.99"],
    "portRanges": [{"low": 80, "high": 80}]
}'

ziti edge create config apache-host.v1 host.v1 '{
    "protocol": "tcp",
    "address": "127.0.0.1",
    "port": 80
}'

ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1 # Did not add an attribute here as you suggested.

ziti edge create service-policy apache-dial-policy Dial \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-private"

ziti edge create service-policy apache-bind-policy Bind \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-public"

```

I used the service name directly in the policy as you suggested:

```bash
ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
      --edge-router-roles '#router-services' \
      --service-roles '@apache-service'

```

* * *

### Results:

#### **Screen Session Log on EC2-Public:**

```auto
About to run tunnel service... ziti-edge-tunnel
(2236)[0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(2236)[0.000] INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2025-01-03T16:13:26.912)
(2236)[0.000] INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(2236)[0.000] INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(2236)[0.000] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(2236)[0.024] INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(2236)[0.025] WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-public.json] is not loaded yet or already removed.
(2236)[0.025] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-public.json]
(2236)[0.025] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-public.json]
(2236)[0.025] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-public.json] loaded
(2236)[0.025] INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(2236)[0.025] INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(2236)[0.025] INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(2236)[0.025] INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(2236)[0.084] INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(2236)[0.098] INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-public] connected to controller
(2236)[0.098] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-public.json] context event : status is OK
(2236)[0.130] INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-public) new channel for ztx[0] identity[EC2-public]
(2236)[0.130] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-public] added edge router router-public@ziti-router-public.example.com
(2236)[0.130] INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(2236)[0.237] INFO tunnel-cbs:ziti_tunnel_ctrl.c:953 on_service() hosting server_address[tcp:127.0.0.1:80] service[apache-service]
(2236)[0.237] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:4Qrx582kuj8EMabc4Y9sSl ===============
(2236)[0.237] INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
(2236)[0.243] WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[0.268] INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
(2236)[0.271] INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(2236)[0.271] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-public] router router-public connected
(2236)[0.271] INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(2236)[0.956] WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[1.093] INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(2236)[2.844] WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[9.507] WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE

```

#### **Screen Session Log on EC2-Private:**

```auto
About to run tunnel service... ziti-edge-tunnel
(1582)[0.000] INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(1582)[0.000] INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2025-01-03T16:10:05.779)
(1582)[0.000] INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(1582)[0.000] INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(1582)[0.000] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(1582)[0.028] WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-private.json] is not loaded yet or already removed.
(1582)[0.028] INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(1582)[0.028] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-private.json]
(1582)[0.028] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-private.json]
(1582)[0.028] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-private.json] loaded
(1582)[0.030] INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(1582)[0.030] INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(1582)[0.030] INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(1582)[0.030] INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(1582)[0.208] INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(1582)[0.228] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:798 on_event() ztx[EC2-private.json] API Event with controller address : (null)
(1582)[0.231] ERROR tunnel-cbs:ziti_tunnel_ctrl.c:1571 update_config_done() updated config file with new URL
(1582)[0.232] INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-private] connected to controller
(1582)[0.232] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-private.json] context event : status is OK
(1582)[0.267] INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-private) new channel for ztx[0] identity[EC2-private]
(1582)[0.267] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-private] added edge router router-private@ziti-router-private.example.com
(1582)[0.267] INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(1582)[0.482] INFO tunnel-cbs:ziti_tunnel_cbs.c:414 new_ziti_intercept() creating intercept for service[apache-service] with intercept.v1 = { "addresses": ["10.100.99.99"], "portRanges": [{ "high": 80, "low": 80 }], "protocols": ["tcp"] }
(1582)[0.482] INFO tunnel-cbs:ziti_tunnel_ctrl.c:938 on_service() starting intercepting for service[apache-service]
(1582)[0.482] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:4Qrx582kuj8EMabc4Y9sSl ===============
(1582)[0.482] INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 2 route updates
(1582)[0.486] INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[2]: 0/OK
(1582)[0.525] INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(1582)[0.525] INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-private] router router-private connected
(1582)[0.525] INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(1582)[1.228] INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected

```

* * *

Here are my current edge router and service configurations:

```auto
$ ziti edge list edge-routers
# Outputs details of all edge routers (router-public, router-private, router-services)
╭────────────┬─────────────────┬────────┬───────────────┬──────┬─────────────────╮
│ ID │ NAME │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├────────────┼─────────────────┼────────┼───────────────┼──────┼─────────────────┤
│ 5TL8AjQpd9 │ router-private │ true │ true │ 0 │ router-private │
│ AQy8Ajppd9 │ router-services │ true │ true │ 0 │ router-services │
│ lgB8NUppw9 │ router-public │ true │ true │ 0 │ router-public │
╰────────────┴─────────────────┴────────┴───────────────┴──────┴─────────────────╯
results: 1-3 of 3

```

```auto
$ ziti edge list service service-edge-router-policies apache-service
# Shows no issues with the service edge router policy.
╭────────────────────────┬─────────────────────────────────────────┬─────────────────┬───────────────────╮
│ ID │ NAME │ SERVICE ROLES │ EDGE ROUTER ROLES │
├────────────────────────┼─────────────────────────────────────────┼─────────────────┼───────────────────┤
│ 30NF1lLnUm0yzylPeSLiLi │ all-services-on-router-services-policy2 │ @apache-service │ #router-services │
╰────────────────────────┴─────────────────────────────────────────┴─────────────────┴───────────────────╯
results: 1-1 of 1

```

Could you please guide me on what might be missing or misconfigured in the current setup?

---

<div class="post-metadata">

### Author: ![scareything](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/scareything/32/106_2.png) [@scareything](https://openziti.discourse.group/u/scareything)
#### Post date: [January 3, 2025, 5:26pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/8 "2025-01-03T17:26:53Z")

</div>

Your identities are only allowed to access `router-private` and `router-public` respectively. These routers are not permitted to carry `apache-service`. The only router that can handle `apache-service` is the new `router-services` router, but I don't see any edge router policies that allow your identities to use the `router-services` router.

It looks like you either need to allow `router-private` and `router-public` to handle `apache-service` with service-edge-router policies, or allow your identities to use the `router-services` router with edge router policies.

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 3, 2025, 5:41pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/9 "2025-01-03T17:41:00Z")

</div>

Ohh, I thought setting up a separate router for services (`router-services`) and dedicated routers for the EC2 instances (`router-private` and `router-public`) would be a better approach to segregate traffic.

But based on your feedback, it seems my way of doing it might not align with how OpenZiti operates, right?

---

<div class="post-metadata">

### Author: ![scareything](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/scareything/32/106_2.png) [@scareything](https://openziti.discourse.group/u/scareything)
#### Post date: [January 3, 2025, 6:12pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/10 "2025-01-03T18:12:10Z")

</div>

A router needs to be accessible by the identity and the service to be a viable path.

> [@am3y](#):
>
> But based on your feedback, it seems my way of doing it might not align with how OpenZiti operates, right?

The only thing that seems odd to me is the way you're creating identities, routers, and (previously) services with a role attribute that matches the name of the entity:

```auto
ziti edge create identity device EC2-private --role-attributes "EC2-private" -o EC2-private.jwt

```

I may be missing something that's important to you, but in my opinion a role-attribute that is only going to be assigned to a single entity is unnecessary, because you can just use the entity name with `@` instead. To me, role attribues are useful when you want to "group" more than one entity together, such as '#public-routers' (plural), or `#boston-users` where you add the attribute to more than one router/identity etc.

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 3, 2025, 6:47pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/11 "2025-01-03T18:47:30Z")

</div>

Thanks for your insights @scareything . Let me clarify the structure I'm working towards, and I'd appreciate your suggestions if my approach needs improvement.

### Current Structure

I'm currently exploring OpenZiti and planning to set it up across multiple EC2 instances divided into zones. Here's how I'm organizing it:

#### Example: Zone-A

- **EC2 Instances:**  
All EC2 instances in Zone-A are within the same VPC as the EKS cluster. For these, I create identities like:

- **Routers:**  
I set up routers with internal-facing load balancers (to ensure traffic between the EC2 instances and the routers stays within the VPC). For example:

- **Router Policies:**  
Any EC2 with the `#EC2-private` attribute can connect to any router with the `#router-private` attribute:

This setup ensures scalability and flexibility since any new EC2 instance or router in Zone-A can just have the appropriate attributes assigned to integrate into the system seamlessly.

* * *

### About Services

I was thinking of replicating a similar approach for services by having a dedicated set of routers (e.g., `router-services`) specifically for handling service traffic.

* * *

### Question

IDK if my approach is right or wrong, but I thought it made sense for scalability and flexibility.

Do you think this is a good way to structure the setup, or would you recommend a different approach? I'm still exploring OpenZiti and would value your suggestions to align with best practices.

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 3, 2025, 6:49pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/12 "2025-01-03T18:49:15Z")

</div>

![openziti2.drawio2](https://global.discourse-cdn.com/free1/uploads/netfoundry/original/2X/8/81712e7bf18875f48abb65d0982afd70cedef9ae.png)

IDK if I'm right or wrong but I was thinking to setup stuff like this

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 5, 2025, 8:30pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/13 "2025-01-05T20:30:44Z")

</div>

Just following up to see if anyone has insights—thanks in advance!

---

<div class="post-metadata">

### Author: ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)
#### Post date: [January 6, 2025, 2:41pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/14 "2025-01-06T14:41:51Z")

</div>

Hi @am3y,

> [@am3y](#):
>
> IDK if my approach is right or wrong, but I thought it made sense for scalability and flexibility.
> 
> Do you think this is a good way to structure the setup, or would you recommend a different approach?

OpenZiti allows you to have a lot of flexibility. Part ot my problem are generic terms like "load balancer" have overloaded meanings. It's generally easier for us to say, "yes that makes sense" from a generic diagram similar to the one you provided.

If I were to look at your diagram, I would change it because as it appears, it doesn't quite make sense to me. The router-public appears to be within private address space, meaning it won't be usable from outside of AWS. Maybe that's what you want, but I don't know for sure. It would be atypical, but maybe that's what you want?

A more typical style of deployment imo would have the controller and the public-router deployed on the right-hand side of your diagram in the 'public' vpc. Then, I would keep one or more routers deployed within the private vpc, and these routers would connect out to the public router(s), forming the secure OpenZiti overlay mesh and providing the (generally) desired reachability from anywhere on the open internet.

That make sense?

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 6, 2025, 2:46pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/15 "2025-01-06T14:46:00Z")

</div>

> [@am3y](#):
>
> Two VPCs are configured: `VPC-public` and `VPC-private`. Note: These public, private are just names for our convinence.

Hii @TheLumberjack  
I guess I have already cleared this in the start. Ignore the public & private thing..

---

<div class="post-metadata">

### Author: ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)
#### Post date: [January 6, 2025, 2:49pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/16 "2025-01-06T14:49:18Z")

</div>

Then it sounds like it'll be just fine, yes. 🙂

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 6, 2025, 3:26pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/17 "2025-01-06T15:26:43Z")

</div>

Yeah but idk why it's not working ☹

---

<div class="post-metadata">

### Author: ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)
#### Post date: [January 6, 2025, 3:30pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/18 "2025-01-06T15:30:21Z")

</div>

I don't understand. As of Jan 2 you reported:

> [@am3y](#):
>
> With these broad policies, I was able to successfully test the service using:
> 
> ```auto
> curl http://10.100.99.99/
> 
> ```

And in the comment I replied to, you had questions, but none of those questions indicated "something wasn't working".

If something isn't working, can you reply with the error and "what" isn't working? I scanned back through and I didn't see it mentioned, but maybe I just missed it.

---

<div class="post-metadata">

### Author: ![am3y](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/am3y/32/2366_2.png) [@am3y](https://openziti.discourse.group/u/am3y)
#### Post date: [January 6, 2025, 3:42pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/19 "2025-01-06T15:42:54Z")

</div>

I thought you have gone through this

> [@Troubleshooting OpenZiti Tunnel: Edge Router and Service Configuration Issues](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/7):
>
> Thanks for your speedy reply @scareything . I tried the following steps as you suggested: ziti edge create edge-router-policy router-private-router-policy \ --edge-router-roles "#router-private" \ --identity-roles "#EC2-private" \ --semantic "AllOf" ziti edge create edge-router-policy router-public-router-policy \ --edge-router-roles "#router-public" \ --identity-roles "#EC2-public" \ --semantic "AllOf" ziti edge create config apache-intercept-ip intercept.v1 '{ "protocols": ["tcp"], …

---

<div class="post-metadata">

### Author: ![TheLumberjack](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/thelumberjack/32/113_2.png) [@TheLumberjack](https://openziti.discourse.group/u/TheLumberjack)
#### Post date: [January 6, 2025, 8:07pm UTC](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683/20 "2025-01-06T20:07:08Z")

</div>

> [@am3y](#):
>
> `NO_EDGE_ROUTERS_AVAILABLE`

Is that the current issue then? Assuming that's the case, I would strongly suggest you revise your service-edge-router-policy (SERP) for the time being and use an `#all/#all` policy. I think the SERP is misconfigured. The service-edge-router-policy object is always hard for people to understand at first and is easy to misconfigure.

A service edge router policy (SERP) indicates the routers a service must enter or exit the openziti overlay from. Looking at your SERP, it looks to me that you have allowed the service to onboard to the overlay on `router-services` only. However, the edge-router policies you created only gives identities access to the two routers (router-private/router-public) which are NOT `router-services`. I think that's the problem.

In general, it's always easier to start open with OpenZiti's policies and refine them to be more granular after you get used to using OpenZiti. I really do suggest starting out with a SERP that's `#all/#all` and remove it from the equation for simplicity's sake. However, I _think_ if you add the other two routers (and remove `router-services` to your policy) it should work.

[Next page](https://openziti.discourse.group/t/troubleshooting-openziti-tunnel-edge-router-and-service-configuration-issues/3683.md?page=2)