Hello, I hope everyone is doing well!
I’m currently testing Open Ziti for simple deployment made of one Windows client with tunneler enabled, one public edge router, one private edge router, and one webserver.
I followed the instructions provided in quickstart in “Host Anywhere Section”. The only difference is that I added on more edge router with private status. Basically I followed the following stepd to deploy private edge router:
- Created a new router on controller and export the jwt;
- Create a router config on the machine hosting the private edge router;
- Edit the yml file with specific information including names and IP;
- Enroll to controller;
Currently I can see all edge router with online and registered status in controller ZAC. After that I created a service to reach a private web server through the private edge router. However I receive the following error log on private edge router:
ERROR edge/router/xgress_edge_tunnel.(*fabricProvider).establishTerminator: {error=[Invalid Session] routerId=[&{0xc0001d3500 kbEB3IzR5W map}] service=[test] address=[f413c773-52e2-4c5b-a08c-a86d0e81dd85]} error creating terminator
In the windows client, I also receive a similar message:
[2022-10-25T15:18:16.452Z] ERROR ziti-sdk:connect.c:941 connect_reply_cb() conn[0.2/Connecting] failed to connect, reason=service 13JuIZW4IgccbwZEaaCm9 has no terminators for instanceId MY IDENTITY NAME
Any clue or tip to troubleshooting and correct this issue?
@Guilherme welcome to the forum and OpenZiti!
Looks like you have made some good progress already. The “error creating terminator” from the private edge router is interesting. Can you run the policy-advisor and see what it tells you?
ziti edge policy-advisor identities
This will tell us a couple things:
- does your router have an identity
- is that router allowed to bind your service
Would you run that and provide the results? I’m also wondering if your router was created with ‘tunneling’ enabled using the -t flag (ziti edge create edge-router ... -t).
Finally, if you run ziti fabric list links do you clearly see the two routers successfully linked? I don’t think this is relevant, just figured I’d make sure you see them linked properly.
Hi @TheLumberjack thanks for your help I've been learning a lot from you in other posts!
Replying your questions, yes I used the following command to deploy the router: "ziti edge create edge-router ip-my-ip.ec2.internal-edge-router -o "${ZITI_HOME}/${router_name}.jwt" -t -a "private""
ziti edge policy-advisor identities:
Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
OKAY : ip-10-10-3-5.ec2.internal-edge-router (2) -> teste (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : Default Admin (2) -> teste (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : Guilherme Ziti (2) -> teste (1) Common Routers: (1/1) Dial: Y Bind: Y
ziti fabric list links:
ziti fabric list links
╭────────────────────────┬───────────────────────────────────────┬─────────────────────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID │ DIALER │ ACCEPTOR │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE │ STATUS │ FULL COST │
├────────────────────────┼───────────────────────────────────────┼─────────────────────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 7JSXxA3vKP6iMs3WBGE0E2 │ ip-10-10-3-5.ec2.internal-edge-router │ ip-10-10-3-241.ec2.internal-edge-router │ 1 │ 3.4ms │ 3.3ms │ Connected │ up │ 7 │
╰────────────────────────┴───────────────────────────────────────┴─────────────────────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
results: 1-1 of 1
The Guilherme Ziti identity, did you intend for that to be able to Bind and dial the same service? I think that might cause the problem you're seeing.
No, I started given access from everyone to everything in order to understanding what was happening. I've just corrected the configuration, but unfortunately still the same error message. Now the new output:
Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
OKAY : ip-10-10-3-5.ec2.internal-edge-router (2) -> teste (1) Common Routers: (1/1) Dial: N Bind: Y
ERROR: Default Admin
- Identity does not have access to any services. Adjust service policies.
OKAY : Guilherme Ziti (2) -> teste (1) Common Routers: (1/1) Dial: Y Bind: N
Alright so I spoke to @plorenz on the matter and he didn’t think this situation was even possible so it feels to me like there’s a bug somewhere. He’s going to dig around in the code looking for holes.
I’ve personally never experienced this issue so if it’s ok, we’re gonna try the “turn it off and on” trick, to get you past this issue. Sound ok? 
So can you turn off both routers, then restart the controller, and then bring both routers back online and see if it happens still?
Hi @Guilherme ,
Can you please share the output of ziti edge ls service edge-routers teste ?
I’m wondering if your service edge router policy for the teste service doesn’t include ip-10-10-3-5.ec2.internal-edge-router. The router identity might have access to a different router, but because that identity is the router, for it host services on that router, the service must have access the router as well.
That sentence was a bit confusing to write, so let me restate it more clearly.
For ip-10-10-3-5.ec2.internal-edge-router to host the service teste, five things must be true:
- The router must be tunneler enabled in the model. This creates an identity for the router with the same name. This lets the router participate in service policies and edge router polices so it can host services. (This looks properly configured in your setup, given your output)
- The router configuration must have the tunnel binding configured, so it’s running the tunneling code. (It does, we can see that in the error message).
- The identity for the router must have bind access to the service (it does)
- The identity for the router must have access to itself, via an edge router policy. (it does, via an autogenerate system policy. The router should always have access to itself).
- The service must have access to the edge router via a service edge router policy. (This is the most likely piece not be there)
Let me know if this helps.
We’re going to have to update the policy advisor to look for this scenario. It’s a tricky one to figure out.
Cheers,
Paul
Ah that makes a lot of sense. Since @Guilherme used the quickstart, it comes with a single attribute on the router as part of the setup script: #public. The wrinkle, is they also stood up that second, private router. We don’t yet have doc that covers this precise use-case (yet, we will). The quickstart makes a blanket “#public/#all” policy. See ziti/ziti-cli-functions.sh at release-next · openziti/ziti · GitHub.
So what you’re describing, @plorenz, makes perfect sense.
To remedy the situation, it sounds like you’ll want to modify this policy. I’m going to go fix the quickstart since it really should be #all/#all to avoid this situation.
@Guilherme - you should be able to run:
ziti edge update service-edge-router-policy allSvcPublicRouters --edge-router-roles '#all' --service-roles '#all'
I’ll go fix the quickstart now too, thanks Paul!
PR is up that fixes this issue. allow all services acess to all routers by default for the quickstart by dovholuknf · Pull Request #876 · openziti/ziti · GitHub