Troubleshooting OpenZiti Tunnel: Edge Router and Service Configuration Issues

Thanks for your speedy reply @scareything . I tried the following steps as you suggested:

ziti edge create edge-router-policy router-private-router-policy \
--edge-router-roles "#router-private" \
--identity-roles "#EC2-private" \
--semantic "AllOf"

ziti edge create edge-router-policy router-public-router-policy \
--edge-router-roles "#router-public" \
--identity-roles "#EC2-public" \
--semantic "AllOf"

ziti edge create config apache-intercept-ip intercept.v1 '{
    "protocols": ["tcp"],
    "addresses": ["10.100.99.99"],
    "portRanges": [{"low": 80, "high": 80}]
}'

ziti edge create config apache-host.v1 host.v1 '{
    "protocol": "tcp",
    "address": "127.0.0.1",
    "port": 80
}'

ziti edge create service apache-service --configs apache-intercept-ip,apache-host.v1 # Did not add an attribute here as you suggested.

ziti edge create service-policy apache-dial-policy Dial \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-private"

ziti edge create service-policy apache-bind-policy Bind \
   --service-roles "@apache-service" \
   --identity-roles "#EC2-public"

I used the service name directly in the policy as you suggested:

ziti edge create service-edge-router-policy all-services-on-router-services-policy2 \
      --edge-router-roles '#router-services' \
      --service-roles '@apache-service'

Results:

Screen Session Log on EC2-Public:

About to run tunnel service... ziti-edge-tunnel
(2236)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(2236)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2025-01-03T16:13:26.912)
(2236)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(2236)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(2236)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(2236)[        0.024]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(2236)[        0.025]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-public.json] is not loaded yet or already removed.
(2236)[        0.025]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-public.json]
(2236)[        0.025]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-public.json]
(2236)[        0.025]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-public.json] loaded
(2236)[        0.025]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(2236)[        0.025]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(2236)[        0.025]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(2236)[        0.025]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(2236)[        0.084]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(2236)[        0.098]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-public] connected to controller
(2236)[        0.098]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-public.json] context event : status is OK
(2236)[        0.130]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-public) new channel for ztx[0] identity[EC2-public]
(2236)[        0.130]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-public] added edge router router-public@ziti-router-public.example.com
(2236)[        0.130]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(2236)[        0.237]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:953 on_service() hosting server_address[tcp:127.0.0.1:80] service[apache-service]
(2236)[        0.237]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:4Qrx582kuj8EMabc4Y9sSl ===============
(2236)[        0.237]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 1 route updates
(2236)[        0.243]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[        0.268]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[1]: 0/OK
(2236)[        0.271]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(2236)[        0.271]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-public] router router-public connected
(2236)[        0.271]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(2236)[        0.956]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[        1.093]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
(2236)[        2.844]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE
(2236)[        9.507]    WARN ziti-sdk:bind.c:246 session_cb() server[0.0](apache-service) failed to get session for service[apache-service]: -17/NO_EDGE_ROUTERS_AVAILABLE

Screen Session Log on EC2-Private:

About to run tunnel service... ziti-edge-tunnel
(1582)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(1582)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.2.1 @g9db50a3(HEAD) starting at (2025-01-03T16:10:05.779)
(1582)[        0.000]    INFO tunnel-sdk:ziti_tunnel.c:60 create_tunneler_ctx() Ziti Tunneler SDK (v1.2.10-beta14)
(1582)[        0.000]    INFO tunnel-cbs:ziti_dns.c:173 seed_dns() DNS configured with range 100.64.0.0 - 100.127.255.255 (4194302 ips)
(1582)[        0.000]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1022 make_socket_path() effective group set to 'ziti' (gid=988)
(1582)[        0.028]    WARN ziti-edge-tunnel:instance.c:39 find_tunnel_identity() Identity ztx[EC2-private.json] is not loaded yet or already removed.
(1582)[        0.028]    INFO ziti-edge-tunnel:resolvers.c:68 init_libsystemd() Initializing libsystemd
(1582)[        0.028]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1135 load_ziti_async() attempting to load ziti instance[EC2-private.json]
(1582)[        0.028]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1142 load_ziti_async() loading ziti instance[EC2-private.json]
(1582)[        0.028]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:425 load_id_cb() identity[EC2-private.json] loaded
(1582)[        0.030]    INFO ziti-sdk:ziti.c:425 ziti_start_internal() ztx[0] enabling Ziti Context
(1582)[        0.030]    INFO ziti-sdk:ziti.c:442 ziti_start_internal() ztx[0] using tlsuv[v0.32.8/OpenSSL 3.3.1 4 Jun 2024]
(1582)[        0.030]    INFO ziti-sdk:ziti_ctrl.c:604 ziti_ctrl_init() ctrl[(null):] using https://ziti-controller.example.com:443/edge/client/v1
(1582)[        0.030]    INFO ziti-sdk:ziti.c:512 ztx_init_controller() ztx[0] Loading ziti context with controller[https://ziti-controller.example.com:443/edge/client/v1]
(1582)[        0.208]    INFO ziti-sdk:ziti.c:1778 version_pre_auth_cb() ztx[0] connected to Legacy controller https://ziti-controller.example.com:443/edge/client/v1 version v1.1.15(0eec47ce3c80 2024-10-02T12:59:41Z)
(1582)[        0.228]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:798 on_event() ztx[EC2-private.json] API Event with controller address : (null)
(1582)[        0.231]   ERROR tunnel-cbs:ziti_tunnel_ctrl.c:1571 update_config_done() updated config file with new URL
(1582)[        0.232]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:981 on_ziti_event() ziti_ctx[EC2-private] connected to controller
(1582)[        0.232]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:463 on_event() ztx[EC2-private.json] context event : status is OK
(1582)[        0.267]    INFO ziti-sdk:channel.c:272 new_ziti_channel() ch[0] (router-private) new channel for ztx[0] identity[EC2-private]
(1582)[        0.267]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1052 on_ziti_event() ztx[EC2-private] added edge router router-private@ziti-router-private.example.com
(1582)[        0.267]    INFO ziti-sdk:channel.c:801 reconnect_channel() ch[0] reconnecting NOW
(1582)[        0.482]    INFO tunnel-cbs:ziti_tunnel_cbs.c:414 new_ziti_intercept() creating intercept for service[apache-service] with intercept.v1 = { "addresses": [ "10.100.99.99" ], "portRanges": [ { "high": 80, "low": 80 } ], "protocols": [ "tcp" ] }
(1582)[        0.482]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:938 on_service() starting intercepting for service[apache-service]
(1582)[        0.482]    INFO ziti-edge-tunnel:ziti-edge-tunnel.c:611 on_event() =============== service event (added) - apache-service:4Qrx582kuj8EMabc4Y9sSl ===============
(1582)[        0.482]    INFO ziti-edge-tunnel:tun.c:196 tun_commit_routes() starting 2 route updates
(1582)[        0.486]    INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[2]: 0/OK
(1582)[        0.525]    INFO ziti-sdk:channel.c:699 hello_reply_cb() ch[0] connected. EdgeRouter version: v1.1.15|0eec47ce3c80|2024-10-02T12:59:41Z|linux|amd64
(1582)[        0.525]    INFO tunnel-cbs:ziti_tunnel_ctrl.c:1056 on_ziti_event() ztx[EC2-private] router router-private connected
(1582)[        0.525]    INFO ziti-edge-tunnel:resolvers.c:402 try_libsystemd_resolver() systemd-resolved selected as DNS resolver manager
(1582)[        1.228]    INFO ziti-sdk:posture.c:206 ziti_send_posture_data() ztx[0] first run or potential controller restart detected

Here are my current edge router and service configurations:

$ ziti edge list edge-routers
# Outputs details of all edge routers (router-public, router-private, router-services)
╭────────────┬─────────────────┬────────┬───────────────┬──────┬─────────────────╮
│ ID         │ NAME            │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES      │
├────────────┼─────────────────┼────────┼───────────────┼──────┼─────────────────┤
│ 5TL8AjQpd9 │ router-private  │ true   │ true          │    0 │ router-private  │
│ AQy8Ajppd9 │ router-services │ true   │ true          │    0 │ router-services │
│ lgB8NUppw9 │ router-public   │ true   │ true          │    0 │ router-public   │
╰────────────┴─────────────────┴────────┴───────────────┴──────┴─────────────────╯
results: 1-3 of 3
$ ziti edge list service service-edge-router-policies apache-service
# Shows no issues with the service edge router policy.
╭────────────────────────┬─────────────────────────────────────────┬─────────────────┬───────────────────╮
│ ID                     │ NAME                                    │ SERVICE ROLES   │ EDGE ROUTER ROLES │
├────────────────────────┼─────────────────────────────────────────┼─────────────────┼───────────────────┤
│ 30NF1lLnUm0yzylPeSLiLi │ all-services-on-router-services-policy2 │ @apache-service │ #router-services  │
╰────────────────────────┴─────────────────────────────────────────┴─────────────────┴───────────────────╯
results: 1-1 of 1

Could you please guide me on what might be missing or misconfigured in the current setup?