I have recently been playing around with OpenZiti. This question is with the free Teams account on NetFoundry and the iOS Ziti Mobile Edge client. I have successfully set up three services. One of them is using specific IPs on a private network and that service works without issue.
I have set up two more that use a private DNS name to access the service. These have issues, but let me get to that. To test that these work fine if I provide their IP address, I added this to the service and I can successfully access it (as expected). It only fails by name.
So, I set logging to TRACE and checked to see what was happening. I see the DNS resolution happen, the request is intercepted as expected and the response was for 100.64.0.26. However, everything stops there. There is no request to access the resource, it just stops. I tried putting in this IP with the same results, nothing happens.
It felt like the IP was not getting routed into the tunnel, so out of curiosity, I went into the advanced settings and changed the NETMASK for the Ziti network to 255.224.0.0 (or /11 instead of the /10). Interestingly, that got both of these other services working. Now, this is on my mobile connection, which might be important here. However, has this come up before? A quick search of the discourse did not turn anything up for me, but I wanted to check. What is the right thing to do here? Thanks.
Hi @mikeshappell, welcome to the community and to OpenZiti!
Often times there are other devices/vpns on a system that end up fighting with one another. You don't have any other "vpn-type" agent installed and running do you? It's common for them to all start at 100.64.0.0. It might be related.
Also virus scanning/endpoint detection software has caused issues in the past where you'll see a connection happen for only a moment, then out of nowhere the connection is terminated.
That's my initial thought/reaction... Others in the community might have other input too.
There are no other active VPN profiles on the device. There is no virus scanning or endpoint detection software loaded either. It is just odd that a small change in the subnet mask makes a difference here. Anyway, curious to see if anyone has ideas why this might be happening for me. Thanks.
UPDATE: Interestingly, a reboot of my phone resolved the problem for the moment. I'm not sure it won't return at some point. I see the pdp_ip0 interface has a route for 100.111.84.159 now.
My guess is that your wireless carrier is giving you an IP from the cgnat range 100.64/10). You should be able to verify that by running ifconfig from LibTerm.
We’ve known for a while that we need to be smarter about getting our own range of IPs for the services that use DNS names, or switch to an ipv6 subnet. As far as I know you are the first person who has actually run into this problem. I’d say for now the best answer we have is to trim down the prefix length to be a better fit for the number of hostnames you plan to use.
Thanks for the response. That was my thought as well, however, on the trimming down the subnet, can I do that safely on the client only? As I am using the NetFoundry controller, will it cause issues elsewhere with something else being assigned an address that does not fit in my subnet? For example, I know the tunneller in Linux shows a /10 as well. Maybe I am missing something here though. Thanks.
The subnet that we are configuring here specifies the range of IPs that will be given out by the tunneler’s internal dns server for hostnames that should be “intercepted”. The dns server (and correspondingly the IPs that it gives out) will only be used locally. The IPs are not synchronized between tunnelers.
So a tunneler on one host may return different IPs than a tunneler on a different host for the same hostnames, even when the tunnelers are intercepting the same services and are configured to use the same subnet for dns mappings.
For this reason the tunnelers can use different prefix lengths, and even different subnets entirely. The only requirements are:
The prefix length must allow for enough IPs to cover the number of hostnames in the intercept configurations that the tunneler is expected to process.
The subnet must not conflict with existing networks that the host is connected to.
I apologize if this response is long-winded. I’m inferring that your question comes from a concern that all tunnelers must use the same IP for a given hostname. Please let me know if I got that wrong, or if you still have any questions.
Thanks for the details. Any chance you can point me to a document or link that contains more details (like this) about how the overall system works? I have read a lot from the site, but nothing I came across had this level of detail or provide more than an overview of the system.
I think in this case, I presumed too much based on my understanding of a different solution that appeared (on the surface) similar, but clearly is much different under the covers. So, any more information you can provide would be appreciated and might even reduce the questions I have :). Thanks.
Unfortunately we haven't documented how the tunnelers work in any detail. It is on the list, but we haven't prioritized it yet. By all means, feel free to ask any questions here in the meantime.
One general point to keep in mind when comparing OpenZiti to VPN solutions is that OpenZiti is more than a VPN. Fundamentally OpenZiti an overlay network with SDKs that applications use to provide and access services. The tunnelers are really just tcp/ip applications that use OpenZiti for the long-haul.
General info about the tunnelers:
The tunnelers generally read packets from a tun device, and manage routes on the host to ensure that packets of interest are dispatched to the tun. The only thing a tunneler knows about an intercepted packet is its source and destination protocol:IP:port. The tunneler tries to match up the destination with one of the OpenZiti services that it knows about, and if one is found it makes a connection for the associated service using the OpenZiti SDK.
So when we talk about "Intercepting" hostnames, it's really just a layer on top of this packet intercept mechanism that dynamically associates the IPs which are mapped in the internal DNS server with the OpenZiti service.
I was actually comparing this in my head with other ZTNA solutions (not a VPN). Some which offer a unique, static IP for a resource across the entire network. That was where I started this question, but I'm with you on the locally assigned addresses which represent an intercepted resource.
I guess I'm interested in understanding how "The tunneler tries to match up the destination with one of the OpenZiti services that it knows about, and if one is found it makes a connection for the associated service using the OpenZiti SDK.". I can make some guesses... but there are a variety of ways I could see this done. Just curious what path was chosen (no pun intended).
There, I finally tricked you into asking about something that we've already written down, at least I think so. Let me know if this doesn't answer your question:
That is good information, but I believe it falls short of my ultimate question (sorry). Let me be more specific using an example from your post. Let's say I have two devices, a Linux tunneller in my network which is bound to a few services (let's use those in the example above). My iPhone is interested in getting to host.ziti.
Based on what I know at the moment, the request is intercepted and assigned an IP from the subnet defined on my iPhone. I guess my question is, as the address assigned is purely local, what happens next. It seems the document you sent is more about what happens when the packets arrive at the Linux machine, or did I miss something here?
The linked document describes how an intercepting tunneler - including Ziti Mobile Edge - decides which service to connect based on the packet that was intercepted. FYI The tunneler-sdk project is used by Ziti Mobile Edge/Ziti Desktop Edge as well as ziti-edge-tunnel (which is used directly by Ziti Desktop Edge for Windows. So most of the tunnelers are using the same codebase as far as packet processing is concerned.
But you're right, the doc doesn't explain what happens once the service is determined. Let me try to explain that with a sequence diagram:
First, the intercepting tunneler (ZME in this case) establishes the hostname to IP mappings for any hostnames in the intercept.v1 address list when it receives the service configurations from the controller.
Wildcard domains (those with a leading '*') are handled a little differently because we don't know what the hostnames will be until we see a query that matches the wildcard domain. So hostname to IP mappings for hostnames in wildcard domains are assigned as the queries are processed by the internal DNS server.
At some point you want to hit "https://host.ziti" from your browser. The browser looks up the hostname (probably using getaddrinfo). If we've set the host's DNS resolver up correctly then the DNS query will be intercepted by ZME, which will give an answer like "100.64.0.3". The browser initiates a connection by sending TCP SYN, and if we've set up a route for that IP to the tun device that ZME is reading then ZME will get the packet and look for the best matching service as described above. Once the service is determined, ZME will "dial" the OpenZiti network for that service.
Previously on another host, a tunneler (ziti-edge-tunnel for the sake of this example) received the same service, but the identity running this tunneler has bind permission so ziti-edge-tunnel is operating as a "hosting" tunneler - meaning that it will accept connections from other OpenZiti identities for the "ziti-hostname" service. When an incoming connection arrives, ziti-edge-tunnel will connect to the address specified in the host.v1 configuration. The status of this connection is reported to the initiating client (ZME in this example).
The intercepting tunneler completes the TCP handshake with the browser only when the hosting tunneler has established a connection on its end. The browser then communicates with ZME as if it was connected with httpd.
Note that only the TCP payloads are sent over the OpenZiti network. The tunnelers on either end of the connection are having separate and distinct conversations with their peers. In other words the tunnelers do not forward the TCP or UDP headers over OpenZiti (as most VPN solutions do).
