No_edge_routers_available

waqas · March 18, 2025, 4:38pm

Hi All, I am working on a POC and this is a simple setup that I am trying to bring up.

I have Controller Deployed via docker compose
I have Router deployed via docker compose
I have User with Laptop on remote location has desktop client installed and registered.
User with Laptop also is registered for service.

I have so far configured ziti-edge-tunnel on the Webserver and identity is registered.
Finally have created the config, service, service-policy for the User with Laptop to connect to Web Server.

Below is a rough diagram of what I am doing. My goal is to allow the user with laptop to nginx-proxy.

However, I am having the following error on ziti-edge-tunnel logs on Webserver

The error is:
(1920426)[ 71435.574] WARN ziti-sdk:bind.c:250 session_cb() server[0.0](nginx-proxy-service) failed to get session for service[nginx-proxy-service]: -17/NO_EDGE_ROUTERS_AVAILABLE

(1920426)[ 71446.642] ERROR ziti-sdk:ziti_ctrl.c:522 ctrl_body_cb() ctrl[dev-controller.amjcan.ca:41280] API request[/sessions] failed code[NO_EDGE_ROUTERS_AVAILABLE] message[No edge routers are assigned and online to handle the requested connection]

ziti edge list edge-routers command output is

╭───────────┬──────────────────────┬────────┬───────────────┬──────┬────────────╮
│ ID        │ NAME                 │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES │
├───────────┼──────────────────────┼────────┼───────────────┼──────┼────────────┤
│ P3qpdhZfJ │router.domain.tld │ true   │ true          │    0 │            │
╰───────────┴──────────────────────┴────────┴───────────────┴──────┴────────────╯

ziti edge list service-policies command output

╭────────────────────────┬─────────────────────┬──────────┬──────────────────────┬────────────────┬─────────────────────╮
│ ID                     │ NAME                │ SEMANTIC │ SERVICE ROLES        │ IDENTITY ROLES │ POSTURE CHECK ROLES │
├────────────────────────┼─────────────────────┼──────────┼──────────────────────┼────────────────┼─────────────────────┤
│ 6K6ZqtMpaMTdRzbeSL3tyb │ nginx-proxy-binding │ AllOf    │ @nginx-proxy-service │ @nginx-proxy   │                     │
│ qzUPdW4XVA6I8BS8Ysy2s  │ nginx-proxy-dialing │ AllOf    │ @nginx-proxy-service │ @waqas         │                     │
╰────────────────────────┴─────────────────────┴──────────┴──────────────────────┴────────────────┴─────────────────────╯

ziti edge list identities command output

╭───────────┬──────────────────────┬─────────┬────────────┬─────────────╮
│ ID        │ NAME                 │ TYPE    │ ATTRIBUTES │ AUTH-POLICY │
├───────────┼──────────────────────┼─────────┼────────────┼─────────────┤
│ 24u.khpkd │ waqas                │ Default │ clients    │ Default     │
│ P3qpdhZfJ │router.domain.tld │ Router  │            │ Default     │
│ fVd30GjS0 │ Default Admin        │ Default │            │ Default     │
│ ob3pFXpfJ │ nginx-proxy          │ Default │ hosts      │ Default     │
╰───────────┴──────────────────────┴─────────┴────────────┴─────────────╯

TheLumberjack · March 18, 2025, 4:45pm

Hi @waqas,

This is a very clear indicator of what's wrong. Thanks for including this in the post. The ziti CLI also has a very useful command:

ziti edge policy-advisor identities -q

This command will show you the list of identities and the services they have assigned as well as if they have routers assigned. If you run this, you'll likely see "0 common routers".

The two policies you're missing right now are "edge-router-policies" and "service-edge-router-policies"

I would recommand you create an edge router policy that states: "all users should be able to use any router annotated as 'public'".

You'd do that with a command similar to this:

ziti edge create edge-router-policy all-ids-public-ers --identity-roles '#all' --edge-router-roles '#public'

You'll also have to pick one (or more) routers to be annotated as "public". Such as:

ziti edge update edge-router public-edge-router -a 'public'

Then, I recommend you start with a service-edge-router-policy that effectively allows any service to use any router:

ziti edge create service-edge-router-policy --service-roles '#all' --edge-router-roles '#all'

I think if you have a peek at these commands and read the doc about what they do (or ask a follow-up if you want), it should hopefully make sense and fix your problem.

Cheers!

EDIT:
A small/quick addendum here. OpenZiti being a zero trust overlay, requires you to authorize identities and services to use routers as well as requiring you to assign services to identities.

waqas · March 18, 2025, 5:40pm

OKAY : waqas (1) -> nginx-proxy-service (1) Common Routers: (1/1) Dial: Y Bind: N 

ERROR: dev-router.amjcan.ca 
  - Identity does not have access to any services. Adjust service policies.

ERROR: Default Admin 
  - Identity does not have access to any services. Adjust service policies.

OKAY : nginx-proxy (1) -> nginx-proxy-service (1) Common Routers: (1/1) Dial: N Bind: Y

TheLumberjack · March 18, 2025, 5:43pm

Is this telling me that you are now successful or that you're still having issues? Some context would be appreciated.

If you're trying to indicate you're still having problems, run policy advisor with services now, instead of identities and confirm services have access to routers as well.

waqas · March 18, 2025, 5:44pm

and when I run this i get this error

error: error updating edge-routers/P3qpdhZfJ instance in Ziti Edge Controller at https://controller.domain.tld:41280/edge/management/v1. Status code: 400 Bad Request, Server returned: {
    "error": {
        "cause": {
            "field": "roleAttributes",
            "reason": "role attributes may not be prefixed with #",
            "value": "#public"
        },
        "code": "COULD_NOT_VALIDATE",
        "message": "The supplied request contains an invalid document or no valid accept content were available, see cause",
        "requestId": "yW8Ibepkd"
    },
    "meta": {
        "apiEnrollmentVersion": "0.0.1",
        "apiVersion": "0.0.1"
    }
}

waqas · March 18, 2025, 5:45pm

Still having issues.

TheLumberjack · March 18, 2025, 5:47pm

i updated my example. sorry about that. your policy advisor indicates to me that your identities appear to have access to a router so you don't need another policy. now we need to verify services have access.

run the policy advisor with services:

ziti edge policy-advisor services -q

and let's see if the services have a policy granting them routers

Can you also run this as well and let's confirm there are policies in place

ziti edge list edge-router-policies
ziti edge list service-edge-router-policies

waqas · March 18, 2025, 5:52pm

I was able to run the previous commands successfully..and here is the output of the commands you just requested.

[user@oz-controller openziti]$ ziti edge policy-advisor services -q
OKAY : waqas (1) -> nginx-proxy-service (1) Common Routers: (1/1) Dial: Y Bind: N 

OKAY : nginx-proxy (1) -> nginx-proxy-service (1) Common Routers: (1/1) Dial: N Bind: Y 

[user@oz-controller openziti]$ 
[user@oz-controller openziti]$ ziti edge list edge-router-policies
╭────────────────────────┬──────────────────────────────┬───────────────────────┬───────────────────────╮
│ ID                     │ NAME                         │ EDGE ROUTER ROLES     │ IDENTITY ROLES        │
├────────────────────────┼──────────────────────────────┼───────────────────────┼───────────────────────┤
│ 4iBG6LyNV0qBUnnVJk0Aqb │ all-endpoints-public-routers │ #public               │ #all                  │
│ 4m1700WJeNH5LOI5LtjNIn │ all-routers                  │ #all                  │ #all                  │
│ P3qpdhZfJ              │ edge-router-P3qpdhZfJ-system │ @dev-router.domain.tld │ @dev-router.domain.tld │
╰────────────────────────┴──────────────────────────────┴───────────────────────┴───────────────────────╯
results: 1-3 of 3
[user@oz-controller openziti]$ 
[user@oz-controller openziti]$ ziti edge list service-edge-router-policies
╭────────────────────────┬─────────────────────────────┬───────────────┬───────────────────╮
│ ID                     │ NAME                        │ SERVICE ROLES │ EDGE ROUTER ROLES │
├────────────────────────┼─────────────────────────────┼───────────────┼───────────────────┤
│ 3lCbVYhlOCDukaXp38E9Kk │ all-routers                 │ #all          │ #all              │
│ 4WpFUfLSgBvd334fvTvHxN │ all-routers-all-services    │ #all          │ #all              │
│ 79ouBQOM6wzTmrjTNPoxHH │ all-service-all-edge-router │ #all          │ #all              │
│ 7EK7Q4hX2j3PqeA4dWaO2D │ nginx-proxy                 │ #rtrhosted    │ #all              │
╰────────────────────────┴─────────────────────────────┴───────────────┴───────────────────╯
results: 1-4 of 4

waqas · March 18, 2025, 5:58pm

The connection is working now...But I'd like to understand what we did and how it worked.
I guess we made edge router a common router and allowed all identities.

I had created these notes to build this connection...
Did I do it right?

Allow USER-LAPTOP to nginx-proxy

Create identies

On controller login via ZITI_PWD=<token> ziti edge login dev-controller.domain.tld:41280 -u admin -p "$ZITI_PWD"


ziti edge create identity user waqas -o user-laptop.jwt

ziti edge create identity nginx-proxy --output-json --jwt-output-file nginx-proxy.jwt

Install Desktop Client

Go to USER-LAPTOP and install Ziti Desktop edge

Copy the user-laptop.jwt file to USER-LAPTOP

Import the identity in the ziti desktop edge client

Server side

On the server side, we need to allow the USER-LAPTOP to connect to the nginx-proxy

We can do this by creating a edge tunneler on the server side.

Make sure the name of the tunneler is the same as the name of the identity that we used while creating the identity. i.e. nginx-proxy


# Download the `ziti-edge-tunnel` file from [here](https://github.com/openziti/ziti-tunnel-sdk-c/releases/latest/).

# Unzip the downloaded file and copy the binary to a directory in your PATH, such as /usr/local/bin.

unzip ziti-edge-tunnel-Linux_x86_64.zip

sudo install -o root -g root ./ziti-edge-tunnel /usr/local/bin/

# Create a POSIX group named ziti if it doesn't already exist. Members of this group can access the identity files and send IPC commands to the socket server.

sudo groupadd --system ziti

# Create a directory to hold the identity files. The tunneler will manage the contents of the identity directory.

sudo mkdir -pv /opt/openziti/etc/identities

# Run the Manually Installed Binary

# When not using the Linux package, you must run the tunneler as root so it will have permission to manage IP routes and the resolver configuration.

sudo chown -cR :ziti /opt/openziti/etc/identities

sudo chmod -cR ug=rwX,o-rwx /opt/openziti/etc/identities

sudo nohup ziti-edge-tunnel run --identity-dir /opt/openziti/etc/identities > out.log 2>&1 &

# Add an Identity

# The tunneller can run with zero or more identities loaded, and needs at least one to make OpenZiti services available on the host. Adding an identity means providing a JWT enrollment token which is used by the tunneller to obtain a client certificate from the OpenZiti controller.

# Root and members of group ziti may add an identity without restarting.

# Get the JWT Token value and run the below command

# sudo ziti-edge-tunnel add -i <IDENTITY> -j "<JWT TOKEN>"

sudo ziti-edge-tunnel add \

-i nginx-proxy \

-j "<JWT-TOKEN>"

Create a zero-trust config

Create a config to allow the USER-LAPTOP to connect to the nginx-proxy


# Config for tunnel to nginx-proxy GUI

# Here host.v1 is the config type and nginx-proxy-gui.v1 is the name of the config.

ziti edge create config nginx-proxy-gui.v1 host.v1 '{"protocol": "tcp","address": "172.21.11.80","port": 40981}'

# Config for intercepter

# Here intercept.v1 is the config type and nginx-proxy-gui-intercept.v1 is the name of the config.

# dev-nginx-proxy.ziti is the FQDN that client will use to connect to the webserver.

ziti edge create config nginx-proxy-gui-intercept.v1 intercept.v1 '{"protocols": ["tcp"],"addresses": ["dev-nginx-proxy.ziti"],"portRanges": [{"low": 9000, "high": 9000}]}'

# Confirm 2 configs are created.

ziti edge list configs

# Config for service that will put together the server to intercepter

ziti edge create service nginx-proxy-service --configs "nginx-proxy-gui-intercept.v1","nginx-proxy-gui.v1"

# Confirm service is created

ziti edge list services

# Now lets create a service bind policy that allows a binding of a client to server.

# Here nginx-proxy-service is the service we created above and the 'nginx-proxy' is the device we created in the beginning.

ziti edge create service-policy nginx-proxy-binding Bind --service-roles '@nginx-proxy-service' --identity-roles '@nginx-proxy'

# Here nginx-proxy-service is the service we created above and the 'waqas' is the user we created in the beginning.

ziti edge create service-policy nginx-proxy-dialing Dial --service-roles '@nginx-proxy-service' --identity-roles '@waqas'

# Update the edge router to accept communication from all identities and make it common/public role.

ziti edge update edge-router dev-router.domain.tld -a 'public'

# Create a service policy to allow the router to accept communication from all identities and make it common/public role.

ziti edge create service-edge-router-policy --service-roles '#all' --edge-router-roles '#all'

TheLumberjack · March 18, 2025, 6:02pm

I'll post this here because I was typing it all up when your "it works now" message came in...

Looking at your notes the one thing that stands out to me is that the service appears to NOT have had the attribute associated to it (i called this out below)

ziti edge create service nginx-proxy-service --configs "nginx-proxy-gui-intercept.v1","nginx-proxy-gui.v1"

I guess I'll stop here since I'm not certain what actually happened. I would still clean those ERPs/SERPs though.

Until you get things working and stable, I would recomend you clean up the ERPs/SERPs.

remove the 'all/all' ERP

ziti edge delete erp 'all-routers'

remove the extra SERPs

ziti edge delete serp 'all-routers'
ziti edge delete serp 'all-service-all-edge-router'
ziti edge delete serp 'nginx-proxy'

That should leave you with two ERPs (the one public/all one and the auto-generated one) and with one SERP (named 'all-routers-all-services' giving all services access to all routers)

The only thing that stands out to me is perhaps that 'nginx-proxy' is taking precedence somehow. The next piece of the puzzle that I don't think you shared yet is the services list and whether the service has the attribute: 'rtrhosted'

If you clean up the ERPs and SERPs and you run policy advisor and you STILL get "no edge routers" we'll keep debugging

waqas · March 18, 2025, 6:16pm

These are my services

╭────────────────────────┬─────────────────────┬────────────┬─────────────────────┬────────────╮
│ ID                     │ NAME                │ ENCRYPTION │ TERMINATOR STRATEGY │ ATTRIBUTES │
│                        │                     │  REQUIRED  │                     │            │
├────────────────────────┼─────────────────────┼────────────┼─────────────────────┼────────────┤
│ 1Pu2nU5MSMcM8NiflwjgSC │ nginx-proxy-service │ true       │ smartrouting        │            │
╰────────────────────────┴─────────────────────┴────────────┴─────────────────────┴────────────╯

Let me digest the rest and get back to you. But thanks for the help so far.

Question before I proceed, how can I not make the router public and be dedicated for the service only?

TheLumberjack · March 18, 2025, 7:38pm

So the "public" annotation is a logical construct. It's what I use to connote that "this router is expected to be used by anyone and is deployed on the public internet". You could call it "internet-access-points" if you prefer, the word "public" has no inherint meaning here.

You need "an annotation" that basically means: "anyone on the open internet with an openziti identity for this overlay network can use routers with this annotation to get onto the openziti overlay network"... That help clarify it?

waqas · March 18, 2025, 7:52pm

Thanks that helps...and now the documentation is also making much more sense to me.
However, I am still confused with #all #public keywords.
My understanding is these are some sort of annotations or attributes as well...but also generally recognized in the system for specific scope.
If so is there a list of these somewhere?

TheLumberjack · March 18, 2025, 8:10pm

The only actual keyword is #all. #public is not a keyword it just -- looks like it could be one. I've struggled with helping people with this exact term for a very long time. It's hard

OpenZiti has one and only one keyword related to policies: #all. Anything else is entirely fabricated and means whatever you want it to mean. You'll also see me use #private to indicate "this router is in private address space. it is not expected that an openziti identity can access this router UNLESS that identity is on the same private network" as well... Just as another example of another "keyword looking" attribute that causes confusion. Terms are hard...

waqas · March 19, 2025, 11:19pm

Thanks Got it.
Today I was able to manipulate access based on attributes.
Thanks for the help.

Topic		Replies	Views
Ziti-service with tunneler	2	27	January 20, 2025
Missing or Invalid CSR General Questions	23	137	May 1, 2024
Ziti Desktop Edge woes	1	20	August 17, 2024
Docker Compose: no ziti edge list service-edge-router-policies configured Ziti Overlay	6	367	January 30, 2023
Invalid edge router for session	2	205	April 17, 2023