Hello all,
I’ve started playing around with OpenZiti and was trying to follow ah “local” deployment (non Docker) using a VM to run the controller and edge router, with two more VM’s in the same subnet, one server and one client. After deploying the controller and router, and then following the guide at Zero Trust Host Access | OpenZiti, I’m seeing that I can indeed curl the crccheck/hello-world container but am doubtful that its running through the overlay. I was not able to see any information indicating that the controller “saw” anything in the openziti.log or router.logs. Furthermore, I was still able to curl the container from my OpenZiti controller VM, which if I understand the framework correctly, shouldn’t be able to see the service. Is there some guidance on how to make sure the setup works and that traffic is going through OpenZiti instead of just the network stack? Thank you
Hi @lzt, welcome to the community!
Skepticism is healthy! 
If you were able to curl to http.ziti you most definitely were going over the overlay. If you were curling to http://localhost:80, you definitely were not going over the overlay. That's the first and easiest way to test.
You should also see on the 'http server' identity in the logs this message (or something very similar):
[ 22.692] INFO tunnel-cbs:ziti_hosting.c:601 on_hosted_client_connect() hosted_service[http.svc], client[http-client] dst_addr[tcp:http.ziti:80]: incoming connection
You can still curl to the container because it this example, it must be left open and exposed via docker so that the ziti-edge-tunnel on the server side can access the port.
Finally, you can use nslookup and verify the http.ziti DNS entry is assigned to an IP in the 100.x.x.x range, like mine here (100.64.0.10 below):
$ nslookup http.ziti
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: http.ziti
Address: 100.64.0.10
You can then try to curl to that IP and you'll see the docker whale as well:
curl 100.64.0.10
<pre>
Hello World
## .
## ## ## ==
## ## ## ## ## ===
/""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o _,/
\ \ _,'
`'--.._\..--''
</pre>
Let me know if that gives you enough information. If not we can keep testing other ways. For example, you can simply turn off one of the ziti-edge-tunnel instances and you will also lose access either from the 'client' side, or from the far/http server side.
Hope that helps.
You can also either turn on events (done in the controller config file). If you turn on events for sessions, circuits, metrics, and usage, that will let you see the session and circuits being created and you can then see the usage data flow out. Alternately, for something a bit quicker you can stream the circuit creation events in realtime using ziti fabric stream circuits.
I think I am missing with the setup because my results are still inconsistent with the guide and your reply here. Is it possible I am using the Quickstart guides incorrectly? My setup:
- openziti-vm
- server-vm
- client-vm
All three of these are on a 172.16.0.0/16 network and can reach each other.
Following the guide, I make sure to start the controller and edge router on openziti-vm, and confirm that the edge router is registered with the controller after zitiLogin and ziti edge list edge-routers
I then go ahead with applying the configs as the are in the guide with the exception of Step 4 where I sub in the IP address for server-vm (On a related note, I also don’t quite understand why the port in this config is set to 8000, shouldn’t it be 80?). For step 7, I sub in http-server for ${http_server_id}based on the listing shown inziti edge list identities`. Finally I move the generated jwts to their respective hosts, server-vm and client-vm.
On server-vm and client-vm I enroll using ziti-edge-tunnel before starting. This is the part where I am getting stuck. On the client-vm I try to curl http.ziti, but it cannot resolve it. I “fix” this by adding an entry in /etc/hosts mapping http.ziti with server-vm IP, but as you said in your comments, I think this does not make traffic go through the overlay. Curl works but I don’t see anything appear in the openziti.log or openziti-edge-router.log. Are there different configurations if I am using multiple machines? It seems like the components connect with each other otherwise.
Thanks again for the help.
Yeah you definitely don't want to do that! Would it be helpful for me to livestream going through this whole process? I'm happy to do that if you would like that? I can do that 'right now' if you like.
I'll go back through the quickstart right now to see if we don't have a bug there. I might have done that guide from the docker-compose point of view. That feels like a bug to me.
Thanks for looking into that. Regarding your offer to livestream, is it possible for me to send you a DM? I wasn’t sure in what format you intended on streaming. Thanks.
I was going to slap it up onto our Youtube channel if you wanted to watch
That sounds great. Thank you.
Thanks for the livestream, sorry I don’t have a Youtube account so can’t comment currently but have identified some stuff that I might need to change, specifically using the identity string versus ‘http_server’. I will give some of these changes a shot and get back to you.
Hope it helped! Definitely found a bug in our setup guide that I’ll go fix now too
I think I’m getting somewhat closer:
(1309)[ 0.090] INFO tunnel-cbs:ziti_dns.c:296 new_ipv4_entry() registered DNS entry http.ziti -> 100.64.0.3
(1309)[ 0.090] INFO tunnel-cbs:ziti_tunnel_ctrl.c:686 on_service() starting intercepting for service[http.svc]
(1309)[ 0.090] INFO ziti-edge-tunnel:ziti-edge-tunnel.c:1262 on_event() =============== service event (added) - http.svc:67HDOpY0qS02ObstDgw6jM ===============
(1309)[ 0.090] INFO ziti-edge-tunnel:tun.c:174 tun_commit_routes() starting 2 route updates
(1309)[ 0.094] INFO ziti-edge-tunnel:tun.c:118 route_updates_done() route updates[2]: 0/OK
(1309)[ 0.102] INFO ziti-sdk:channel.c:640 hello_reply_cb() ch[0] connected. EdgeRouter version: v0.26.11|807dd591b1f5|2022-11-10T14:53:29Z|linux|amd64
(1309)[ 0.102] INFO tunnel-cbs:ziti_tunnel_ctrl.c:801 on_ziti_event() ztx[http-client] router openziti-edge-router@tls://openziti:3022 connected
(1309)[ 1.068] INFO ziti-sdk:posture.c:204 ziti_send_posture_data() ztx[0] first run or potential controller restart detected
http.ziti
Server: 127.0.0.53
Address: 127.0.0.53#53
** server can't find http.ziti: NXDOMAIN
This is on my client side. I see that it looks like http.ziti was registered but when I nslookup http.ziti I’m still not getting anything back - is there something else I may have needed to add?
That makes me think your client didn’t get authorized properly. Can you try running this ziti CLI command?
# see below: ziti edge policy-advisor identities -q
Oh actually - I see this client should have an entry in your log (I scrolled right):
http.ziti -> 100.64.0.3
What OS is your client running? I don’t think you mentioned that yet. I haven’t seen this particular issue in a while. I’ll have to talk to some people to see if anyone has a thought here. I’ve not seen the intercepts fail with NXDOMAIN in a while and I’m not the linux-whiz when it comes to DNS.
Sorry I didn’t mention earlier, I’m running Ubuntu 18.04. Running the command I get the following:
ERROR: openziti-edge-router
- Identity does not have access to any services. Adjust service policies.
OKAY : http-client (1) -> http.svc (1) Common Routers: (1/1) Dial: Y Bind: N
ERROR: Default Admin
- Identity does not have access to any services. Adjust service policies.
OKAY : http-server (1) -> http.svc (1) Common Routers: (1/1) Dial: N Bind: Y
Should I delete the identities and try to remake those as well as the services again?
Nope. That all looks perfect. I’m not exactly sure why you’re getting NXDOMAIN like that. It’s almost as though the ziti resolver isn’t getting first crack at the DNS request. I definitely would expect your nslookup to return 100.64.0.3 for http.ziti…
I’ve not seen this fail for a very long time. And you’re running the latest ziti-edge-tunnel I expect?
I am using the 18.04 version provided in the Tunnelers page for Linux.
You can try to reach “100.64.0.3”… if that works, then your open ziti is working as far as the path is concerned. Then we are left with question about the DNS resolution.
Just tried and I’m unfortunately unable to curl 100.64.0.3 from my client VM.
Is it okay for you to attach the .json file plus the log from the intercepting / dialing side.
Thanks.
Which json file, @JamminSoleng ? Not the identity file, right? That'll have a private key inside it. We don't want that, I'm not sure what other json file to attach.
I am definitely interested in the logs from the intercepting side though, that's a good call. Can you run the 'client' side like this?
./ziti-edge-tunnel run -i client.id.here.json -v6