Understand service-policy behavior in 1 controller + 2 router topology

I am experimenting with a simple topology that has 1 controller and 2 routers (all on a single WSL2 VM).

One router is a "public" router and runs the edge service on primary-interface:8442 and a link listener on 127.0.0.1:10080. It has the role attribute public-routers.

The other router is a "private router" that runs the edge service on 127.0.0.1:9442 and runs a tunneller in host mode.

A simple http server is running on 127.0.0.1:80 (named wpad.host.zitipoc.com in the /etc/hosts file).

The service configuration is as follows:

./bin/ziti edge create config wpad-intercept-v1 intercept.v1 \
    '{"protocols":["tcp"],"addresses":["wpad.host.zitipoc.com"], "portRanges":[{"low":80, "high":80}]}'

./bin/ziti edge create config wpad-host-v1 host.v1 \
    '{"forwardProtocol": true, "allowedProtocols": ["tcp"], "forwardAddress": true, "allowedAddresses": ["wpad.host.zitipoc.com"], "forwardPort": true, "allowedPortRanges": [{"low" : 80, "high": 80}]}'

./bin/ziti edge create service wpad \
    --configs wpad-intercept-v1,wpad-host-v1 \
    --role-attributes wpad

./bin/ziti edge create service-policy wpad-policy-dial Dial \
    --service-roles "#wpad" --identity-roles '#proxy-clients' --semantic AnyOf

./bin/ziti edge create service-policy wpad-policy-bind Bind \
    --service-roles '#wpad' --identity-roles "#private-routers" --semantic AnyOf

./bin/ziti edge create service-edge-router-policy wpad-service-edge-router-policy \
    --service-roles '#wpad' --edge-router-roles '#public-routers,#private-routers' --semantic AnyOf

./bin/ziti edge create edge-router-policy proxy-clients-edge-router-policy \
    --identity-roles '#proxy-clients' --edge-router-roles '#public-routers' --semantic AnyOf

This configuration does not work. There are no terminators created. The policy advisor complains:

$ ./bin/ziti edge policy-advisor identities
...
ERROR: <private-router-name>
  - Identity does not have access to any services. Adjust service policies.

If I change the bind policy to the following, then things start working (and 2 terminators are created, one for edge using the public router, and other for tunnel using the private router):

./bin/ziti edge create service-policy wpad-policy-bind Bind \
    --service-roles '#wpad' --identity-roles "#all" --semantic AnyOf

I am unable to understand why the #all identity-role is required in the bind policy. I would only expect to include the routers running the tunneller in the bind policy.

Can you issue:

ziti edge list identities 'name contains "router"'

Let's ensure the router identity has the "private-rotuers" attribute associated to it. My expectation is that it does not. :slight_smile:

Sorry for the delayed response @TheLumberjack .
The routers have the correct role attributes:

$ ./bin/ziti edge list edge-routers
╭────────────┬──────────────────────────────────────────────────┬────────┬───────────────┬──────┬─────────────────╮
│ ID         │ NAME                                             │ ONLINE │ ALLOW TRANSIT │ COST │ ATTRIBUTES      │
├────────────┼──────────────────────────────────────────────────┼────────┼───────────────┼──────┼─────────────────┤
│ 4yHNeOROmE │ eeed869bb2744a25aa1c659ecca73f04.pub.zitipoc.com │ true   │ true          │    0 │ public-routers  │
│ x2y1eOiOmE │ 99350c34a79e11ef993400155de68372.pub.zitipoc.com │ true   │ true          │    0 │ private-routers │
╰────────────┴──────────────────────────────────────────────────┴────────┴───────────────┴──────┴─────────────────╯
results: 1-2 of 2

$ ./bin/ziti edge list identities
╭────────────┬──────────────────────────────────────────────────┬─────────┬───────────────┬─────────────╮
│ ID         │ NAME                                             │ TYPE    │ ATTRIBUTES    │ AUTH-POLICY │
├────────────┼──────────────────────────────────────────────────┼─────────┼───────────────┼─────────────┤
│ 0cbXhXb9A  │ Default Admin                                    │ Default │               │ Default     │
│ w9r3-ORTB  │ proxy-client                                     │ Default │ proxy-clients │ Default     │
│ x2y1eOiOmE │ 99350c34a79e11ef993400155de68372.pub.zitipoc.com │ Router  │               │ Default     │
╰────────────┴──────────────────────────────────────────────────┴─────────┴───────────────┴─────────────╯

Can you list the edge router policies and service edge router policies and identities please too? Just to get all the data close to each other

Sure

$ ./bin/ziti edge list configs 'name contains "wpad"'
╭────────────────────────┬───────────────────┬──────────────╮
│ ID                     │ NAME              │ CONFIG TYPE  │
├────────────────────────┼───────────────────┼──────────────┤
│ 27QYl3NE84PthXD3AShBK3 │ wpad-host-v1      │ host.v1      │
│ 42cLOAKsTV8Um2OT5iBjni │ wpad-intercept-v1 │ intercept.v1 │
╰────────────────────────┴───────────────────┴──────────────╯
results: 1-2 of 2


$ ./bin/ziti edge list services 'name contains "wpad"'
╭────────────────────────┬──────┬────────────┬─────────────────────┬────────────╮
│ ID                     │ NAME │ ENCRYPTION │ TERMINATOR STRATEGY │ ATTRIBUTES │
│                        │      │  REQUIRED  │                     │            │
├────────────────────────┼──────┼────────────┼─────────────────────┼────────────┤
│ 7Y5aJmxILPNiZTI78vMF2a │ wpad │ true       │ smartrouting        │ wpad       │
╰────────────────────────┴──────┴────────────┴─────────────────────┴────────────╯
results: 1-1 of 1

$ ./bin/ziti edge list service-policies 'name contains "wpad"'
╭────────────────────────┬──────────────────┬──────────┬───────────────┬──────────────────┬─────────────────────╮
│ ID                     │ NAME             │ SEMANTIC │ SERVICE ROLES │ IDENTITY ROLES   │ POSTURE CHECK ROLES │
├────────────────────────┼──────────────────┼──────────┼───────────────┼──────────────────┼─────────────────────┤
│ 5AEjaskkRFuXEUEzTjo0tt │ wpad-policy-bind │ AnyOf    │ #wpad         │ #private-routers │                     │
│ 6q5JiBCY8JtFNklnWjD2Y8 │ wpad-policy-dial │ AnyOf    │ #wpad         │ #proxy-clients   │                     │
╰────────────────────────┴──────────────────┴──────────┴───────────────┴──────────────────┴─────────────────────╯
results: 1-2 of 2

$ ./bin/ziti edge list service-edge-router-policies 'name contains "wpad"'
╭────────────────────────┬─────────────────────────────────┬───────────────┬──────────────────────────────────╮
│ ID                     │ NAME                            │ SERVICE ROLES │ EDGE ROUTER ROLES                │
├────────────────────────┼─────────────────────────────────┼───────────────┼──────────────────────────────────┤
│ 7e5dt20A3Yl7XvXZExkIsO │ wpad-service-edge-router-policy │ #wpad         │ #private-routers #public-routers │
╰────────────────────────┴─────────────────────────────────┴───────────────┴──────────────────────────────────╯
results: 1-1 of 1

$ ./bin/ziti edge list edge-router-policies
╭────────────────────────┬──────────────────────────────────┬───────────────────────────────────────────────────┬───────────────────────────────────────────────────╮
│ ID                     │ NAME                             │ EDGE ROUTER ROLES                                 │ IDENTITY ROLES                                    │
├────────────────────────┼──────────────────────────────────┼───────────────────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 6Bt6UzZBArlyV2VNTvmvCN │ proxy-clients-edge-router-policy │ #public-routers                                   │ #proxy-clients                                    │
│ x2y1eOiOmE             │ edge-router-x2y1eOiOmE-system    │ @99350c34a79e11ef993400155de68372.pub.zitipoc.com │ @99350c34a79e11ef993400155de68372.pub.zitipoc.com │
╰────────────────────────┴──────────────────────────────────┴───────────────────────────────────────────────────┴───────────────────────────────────────────────────╯
results: 1-2 of 2

So the EASIEST thing to do, not the 'best' is to just make two blanket all polices and run policy advisor and see if that 'solves' the problem ... I'd imagine it will...

ziti edge create edge-router-policy blanket-erp --edge-router-roles '#all' --identity-roles '#all'
ziti edge create service-edge-router-policy blanket-serp --edge-router-roles '#all' --service-roles '#all'

ASSUMING that fixes it, well, then we can remove one, then the other to see which one is necessary. I expect the edge-router-policy is somehow mistaken

@TheLumberjack
So to get this working what I have to do is to change the service bind policy to the following:

./bin/ziti edge update service-policy wpad-policy-bind --identity-roles '#all'

With that change the terminators get created.

and I assume you want the router to do the bind, right? So if you run:

./bin/ziti edge update service-policy wpad-policy-bind --identity-roles '@99350c34a79e11ef993400155de68372.pub.zitipoc.com' --service-roles '#wpad'

Do the terminators get created?

With the --identity-roles '@99350c34a79e11ef993400155de68372.pub.zitipoc.com' or --identity-roles '#all' the terminator gets created.

$ ./bin/ziti edge list terminators
╭────────────────────────┬─────────┬──────────────────────────────────────────────────┬─────────┬────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID                     │ SERVICE │ ROUTER                                           │ BINDING │ ADDRESS                │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├────────────────────────┼─────────┼──────────────────────────────────────────────────┼─────────┼────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ 4BN1eEaMd6kuWbUky7SF94 │ wpad    │ 99350c34a79e11ef993400155de68372.pub.zitipoc.com │ tunnel  │ 4BN1eEaMd6kuWbUky7SF94 │          │    0 │ default    │            0 │
╰────────────────────────┴─────────┴──────────────────────────────────────────────────┴─────────┴────────────────────────┴──────────┴──────┴────────────┴──────────────╯
results: 1-1 of 1

But with --identity-roles '#private-routers' on the bind service-policy the terminator does not get created:

$ ./bin/ziti edge list service-policies 'name contains "wpad"'
╭────────────────────────┬──────────────────┬──────────┬───────────────┬──────────────────┬─────────────────────╮
│ ID                     │ NAME             │ SEMANTIC │ SERVICE ROLES │ IDENTITY ROLES   │ POSTURE CHECK ROLES │
├────────────────────────┼──────────────────┼──────────┼───────────────┼──────────────────┼─────────────────────┤
│ 5AEjaskkRFuXEUEzTjo0tt │ wpad-policy-bind │ AnyOf    │ #wpad         │ #private-routers │                     │
│ 6q5JiBCY8JtFNklnWjD2Y8 │ wpad-policy-dial │ AnyOf    │ #wpad         │ #proxy-clients   │                     │
╰────────────────────────┴──────────────────┴──────────┴───────────────┴──────────────────┴─────────────────────╯
results: 1-2 of 2

$ ./bin/ziti edge list terminators
╭────┬─────────┬────────┬─────────┬─────────┬──────────┬──────┬────────────┬──────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├────┼─────────┼────────┼─────────┼─────────┼──────────┼──────┼────────────┼──────────────┤
╰────┴─────────┴────────┴─────────┴─────────┴──────────┴──────┴────────────┴──────────────╯
results: none

I'm sorry, I think your post on terminators before got me a little sidetracked. Does policy advisor show you "common routers" now or are there still 0/0?

Yes that's what I would expect. It's really, really easy to mix up edge router name and edge router identity because they are identically named. However, for that to work you will need to update the edge router identity with #private-routers.

Right now, your router identity has no attributes:

$ ./bin/ziti edge list identities
╭────────────┬──────────────────────────────────────────────────┬─────────┬───────────────┬─────────────╮
│ ID         │ NAME                                             │ TYPE    │ ATTRIBUTES    │ AUTH-POLICY │
├────────────┼──────────────────────────────────────────────────┼─────────┼───────────────┼─────────────┤
│ 0cbXhXb9A  │ Default Admin                                    │ Default │               │ Default     │
│ w9r3-ORTB  │ proxy-client                                     │ Default │ proxy-clients │ Default     │
│ x2y1eOiOmE │ 99350c34a79e11ef993400155de68372.pub.zitipoc.com │ Router  │               │ Default     │
╰────────────┴──────────────────────────────────────────────────┴─────────┴───────────────┴─────────────╯

So I created the router using the following command, which also created the edge-router identity:

./bin/ziti edge create edge-router 99350c34a79e11ef993400155de68372.pub.zitipoc.com \
    --jwt-output-file ./etc/99350c34a79e11ef993400155de68372.jwt \
    --tunneler-enabled \
    --role-attributes 'private-routers'

So I am confused as to why the edge-router identity did not get the role-attribute.

That's quite understandable. It might be something we consider adding a second flag for maybe. The command shown adds attributes to the router, not to the identity so that you can then use that attribute in edge router policies or service edge router policies...

--tunneler-enabled creates an identity, but it does not update the identity with the same attribute as the router. Perhaps we'll add a --tunneler-attribute type of flag in the future, but for now you just need to update the identity with whatever attribute you wish.

Ok @TheLumberjack .

I just updated the identity and the terminator shows up now:

$ ./bin/ziti edge update identity 99350c34a79e11ef993400155de68372.pub.zitipoc.com --role-attributes 'private-routers'

$ ./bin/ziti edge list terminators
╭────────────────────────┬─────────┬──────────────────────────────────────────────────┬─────────┬────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID                     │ SERVICE │ ROUTER                                           │ BINDING │ ADDRESS                │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├────────────────────────┼─────────┼──────────────────────────────────────────────────┼─────────┼────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ 7JbZ01xLB4WPakl07h78NV │ wpad    │ 99350c34a79e11ef993400155de68372.pub.zitipoc.com │ tunnel  │ 7JbZ01xLB4WPakl07h78NV │          │    0 │ default    │            0 │
╰────────────────────────┴─────────┴──────────────────────────────────────────────────┴─────────┴────────────────────────┴──────────┴──────┴────────────┴──────────────╯
results: 1-1 of 1

Thanks a lot for your help as always!