I'm doing a PoC to test out tunnelling feature of OpenZiti, the setup follows on Ziti-Edge-Router as Gateway | OpenZiti, except that I don't have the public network, and I only have two private subnet 10.0.1.0/24 and 10.0.2.0/24. I want to test tunnelling feature which allow me to access service running on 8000 in subnet 1 from subnet 2 via port 443.
I follow the instructions and setup the ziti network and routers successfully. The setup is
If i test on 10.0.2.11 which runs router instances, i was able to access httt service via 80 "curl http://10.0.1.100"
Do the same test on 10.0.2.100, and add static route gateway to 10.0.2.11 for dest 10.0.1.100, i'm not able to access either http://10.0.1.100 and http://10.0.1.100:8000, there is no any logs at edge router and controller
Is this kind of LAN setup workable (There is no Openziti network running on public network)?
If this is a workable scenario, how to troubleshooting on client (10.0.2.100) that it can't access http service when static route is set? In this case there is no any log on edge router and controller.
Hi @cao. Welcome to the community and to OpenZiti.
OpenZiti has no requirement for public network at all. You simply need to configure the network properly and the pieces of the overlay need to be able to communicate to form links or for the edge clients to connect to edge routers.
You should first verify that the routers are able to form a link. Run ziti fabirc list links and confirm you see one link like this:
ziti fabric list links
╭────────────────────────┬──────────────────────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID │ DIALER │ ACCEPTOR │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE │ STATUS │ FULL COST │
├────────────────────────┼──────────────────────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 6nx2dJrgRXOnkiLzzYBU6m │ ip-172-31-11-231-edge-router │ 14pp │ 1 │ 65000.0ms │ 65000.0ms │ Connected │ up │ 130001 │
╰────────────────────────┴──────────────────────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
Once you have that one link, then you should probably turn on verbose debugging to get more details from the router.
I would expect once you have the link setup, the next question will be confirming your services are setup correctly.
@JamminSoleng is this something you could help with?
@TheLumberjack
Thank you very much for the quick advice.
I run the command to check the link, it shows good
ziti fabric list links**0**
╭────────────────────────┬─────────────────┬──────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────╮
│ ID │ DIALER │ ACCEPTOR │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE │ STATUS │ FULL COST │
├────────────────────────┼─────────────────┼──────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┤
│ 2Vq4lH5FeU4ni58i2HJpN7 │ clients-router │ ip-10-0-1-10-edge-router │ 1 │ 2.7ms │ 2.6ms │ Connected │ up │ 5 │
│ 3gatcE3LejfChBvKY79O9f │ hosts-router │ ip-10-0-1-10-edge-router │ 1 │ 2.0ms │ 2.8ms │ Connected │ up │ 5 │
╰────────────────────────┴─────────────────┴──────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────╯
with both clients-router and hosts-router run with verbose, nothing be print out, when i visit "curl http://10.0.1.100" from 10.0.2.100
I believe the configuration is somewhat correct, because if i run the same cmd "curl http://10.0.1.100" from 10.0.2.11. (this host have edge router cleints-router running on it), it works well and i can see logs on both routers.
Question:
Given that same command works on 10.0.2.11, BUT it doesn't work on 10.0.2.100, beside that i need to set a static route on 10.0.2.100 (ip route add 10.0.1.100/32 via 10.0.2.11), any other config needed?
Even edge router already run in verbose, nothing be print out when do "curl http://10.0.1.100" on 10.0.2.100, any suggestion how i can do troubleshoot?
There are few things you can check for us to see where the connection problems are.
First, the local-er (10.0.2.11) will need to be able to connect to the controller at 10.0.1.10. So I assume you have some kind static route setup on local-er to reach controller. Did you setup route to 10.0.1.10/32 or 10.0.1.0/24? if you setup for the whole subnet, then that VM (local-er) will be able to access 10.0.1.100 without ziti.
Second, the firewall on the local-er will need to open to access traffic on port 80 (curl/http traffic). Please check on that by doing: sudo ufw status
Third, can you go to controller, and print the following output:
ziti edge list terminators
ziti edge list services
ziti edge list identities
ziti edge list service-policies
ziti edge list service-edge-router-policies
ziti edge policy-advisor services
Thanks for your suggestion, really good point to tcpdump on near end router 10.0.2.11. BUT nothing captured. Here is what I do.
Double check the static route on 10.0.2.100
ubuntu@ip-10-0-2-100:~$ ip route s
default via 10.0.2.1 dev eth0 proto dhcp src 10.0.2.100 metric 100
10.0.0.2 via 10.0.2.1 dev eth0 proto dhcp src 10.0.2.100 metric 100
10.0.1.100 via 10.0.2.11 dev eth0
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.100 metric 100
10.0.2.1 dev eth0 proto dhcp scope link src 10.0.2.100 metric 100
Run tcpdump on 10.0.2.11 to check if any packets from/to 10.0.2.100
sudo tcpdump host 10.0.2.100
Run "curl http://10.0.1.100" on 10.0.2.100, BUT nothing captured in tcpdump in the previous step.
Does that means static routing is NOT working, what other steps can do to troubleshooting why data is not reach the near end router (local-router) 10.0.2.11?
Additional notes: If i run "nslookup mysimpleservice.ziti" on 10.0.2.100, i do see dns query be captured by tcpdump.
Yes, for sure local-er (10.0.2.11) can connect to controller.(As i mentioned earlier, when i "curl http://10.0.1.100" from host 10.0.2.11, tunneling works as expected)
Without Ziti, Subnet 10.0.1.0/24 and 10.0.2.0/24 can access each other.
Some behavior i observed when run testing on 10.0.2.11 (local-router).
I didn't even enable the firewall on any of hosts in this testing to make troubleshooting easy.
'''
ubuntu@ip-10-0-2-11:~$ sudo ufw status
Status: inactive
'''
Here is the output, kindly help me double check if anything is mis-configed.
(I'm sorry I have no idea how to make table display properly. Table in my first post display well, but here is a bit messy. Any hints to format the table, i can edit it to make it more readable.)
ubuntu@ip-10-0-1-10:~$ ziti edge list terminators
╭──────────────────────────────────────┬─────────────────┬───────────────┬─────────┬──────────────────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├──────────────────────────────────────┼─────────────────┼───────────────┼─────────┼──────────────────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ 1a692ba3-5306-4a0d-a53d-c3c661a05206 │ tcp8000-service │ hosts-router │ tunnel │ 1a692ba3-5306-4a0d-a53d-c3c661a05206 │ │ 0 │ default │ 0 │
│ 98865ffb-ad53-47fa-ba27-edc276cd9f7d │ tcp80-service │ hosts-router │ tunnel │ 98865ffb-ad53-47fa-ba27-edc276cd9f7d │ │ 0 │ default │ 0 │
╰──────────────────────────────────────┴─────────────────┴───────────────┴─────────┴──────────────────────────────────────┴──────────┴──────┴────────────┴──────────────╯
buntu@ip-10-0-1-10:~$ ziti edge list services
╭────────────────────────┬─────────────────┬────────────┬─────────────────────┬────────────╮
│ ID │ NAME │ ENCRYPTION │ TERMINATOR STRATEGY │ ATTRIBUTES │
│ │ │ REQUIRED │ │ │
├────────────────────────┼─────────────────┼────────────┼─────────────────────┼────────────┤
│ 7Mg7LQD9Xm7u4X5Q8Hx9R1 │ tcp8000-service │ true │ smartrouting │ rtrhosted │
│ lrN20ygSUZYjFhqf5x9rR │ tcp80-service │ true │ smartrouting │ rtrhosted │
╰────────────────────────┴─────────────────┴────────────┴─────────────────────┴────────────╯
ubuntu@ip-10-0-1-10:~$ ziti edge list identities
╭────────────┬──────────────────────────┬─────────┬────────────┬─────────────╮
│ ID │ NAME │ TYPE │ ATTRIBUTES │ AUTH-POLICY │
├────────────┼──────────────────────────┼─────────┼────────────┼─────────────┤
│ 9u8MwmS9G4 │ clients-router │ Router │ clients │ Default │
│ X5mlChnPUx │ ip-10-0-1-10-edge-router │ Router │ │ Default │
│ XQKCUmKl1 │ hosts-router │ Router │ hosts │ Default │
│ wNN1g9puf │ Default Admin │ Default │ │ Default │
╰────────────┴──────────────────────────┴─────────┴────────────┴─────────────╯
ubuntu@ip-10-0-1-10:~$ ziti edge list service-policies
╭────────────────────────┬─────────────┬──────────┬───────────────┬────────────────┬─────────────────────╮
│ ID │ NAME │ SEMANTIC │ SERVICE ROLES │ IDENTITY ROLES │ POSTURE CHECK ROLES │
├────────────────────────┼─────────────┼──────────┼───────────────┼────────────────┼─────────────────────┤
│ 40IykpiJAHtqUAxCnvQttn │ bind-policy │ AnyOf │ #rtrhosted │ #hosts │ │
│ 7cOwx3EkRXnoG4BUQLhHad │ dial-policy │ AnyOf │ #rtrhosted │ #clients │ │
╰────────────────────────┴─────────────┴──────────┴───────────────┴────────────────┴─────────────────────╯
ubuntu@ip-10-0-1-10:~$ ziti edge list service-edge-router-policies
╭────────────────────────┬──────────────────┬───────────────┬───────────────────╮
│ ID │ NAME │ SERVICE ROLES │ EDGE ROUTER ROLES │
├────────────────────────┼──────────────────┼───────────────┼───────────────────┤
│ 5F8ameZPjIDLEzez4KZtYP │ allSvcAllRouters │ #all │ #all │
╰────────────────────────┴──────────────────┴───────────────┴───────────────────╯
buntu@ip-10-0-1-10:~$ ziti edge policy-advisor services
Policy General Guidelines
In order for an identity to dial or bind a service, the following must be true:
- The identity must have access to the service via a service policy of the correct type (dial or bind)
- The identity must have access to at least one on-line edge router via an edge router policy
- The service must have access to at least one on-line edge router via a service edge router policy
- There must be at least one on-line edge router that both the identity and service have access to.
Policy Advisor Output Guide:
STATUS = The status of the identity -> service reachability. Will be OKAY or ERROR.
ID = identity name
ID ROUTERS = number of routers accessible to the identity via edge router policies.
- See edge router polices for an identity: ziti edge controller list identity edge-router-policies <identity>
SVC = service name
SVC ROUTERS = number of routers accessible to the service via service edge router policies.
- See service edge router policies for a service with: ziti edge controller list service service-edge-router-policies <service>
ONLINE COMMON ROUTERS = number of routers the identity and service have in common which are online.
COMMON ROUTERS = number of routers (online or offline) the identity and service have in common.
DIAL_OK = indicates if the identity has permission to dial the service.
- See service polices for a service : ziti edge controller list service service-policies <service>
- See service polices for an identity: ziti edge controller list identity service-policies <identity>
BIND_OK = indicates if the identity has permission to bind the service.
ERROR_LIST = if the status is ERROR, error details will be listed on the following lines
Output format: STATUS: ID (ID ROUTERS) -> SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
-------------------------------------------------------------------------------
OKAY : clients-router (2) -> tcp8000-service (3) Common Routers: (2/2) Dial: Y Bind: N
OKAY : hosts-router (2) -> tcp8000-service (3) Common Routers: (2/2) Dial: N Bind: Y
OKAY : clients-router (2) -> tcp80-service (3) Common Routers: (2/2) Dial: Y Bind: N
OKAY : hosts-router (2) -> tcp80-service (3) Common Routers: (2/2) Dial: N Bind: Y
My issue is related to AWS EC2 instance config, I'm using AWS ec2 as PoC environment, there is tricky that i have to turn off EC2 instance's Network source/destination check in order to let local-router (10.0.2.11) receive and forward the traffic.
I took liberty and edited your comment and added triple tick fences. Discourse supports markdown. It looks like this:. You put them above and below the stuff you want in the block. Also see Post code or preformatted text - users - Discourse Meta
But if you are running openziti off public cloud, you can checkout our public cloud section. We have specifically mentioned about the source and destination check.
