Source IP / Network segmention feature

Hi All,

The last week, I tested everything Open Ziti has to offer. I created my first network, zitified Python and C application, lan-gateways with the tunneler and the Router. I also tested Cloud-to-Cloud connectivity and several approaches from the network side.
I do understand the idea, and honestly I like it a lot. Also, the DNS part seems to be a game changer in the network world.
I really like this technology!

But I have one problem which does impact the implementation a lot.

From a client/implementer perspective, one thing is missing from my point of view. To generate, a full-blown ZeroTrust overlay network, you need a lot of time, effort, and processes in place to get it working. You require server-, client-, SRE-, and developer teams working together how to approach and implement this solution based on their needs. Most of the time, this will can't be done from the ground up. It's a large, quite big project which can run a year or longer (depending on size).

During the time of creation, you need to start with the location and or VLANs/Vnets/VPCs with LAN-gateways. Other options seem not executable, as you would need to have one "Big-Bang" implementation, where everything needs to be zero-trust, up and running from Go-Live onwards.

This is not an option at all for a medium to big sized company.

Problem Description (hypothetical):

I tested this approach with the LAN Gateways based on router level. I played a lot with the configs and DNS settings, everything seems fine but one thing. If I have ONE router installed, let's say at a location. There are multiple VLANS:

  • Vlan1: Guest Network --> Should have no OpenZiti Connection
  • Vlan2: Office Network --> should have access from Identity SideA.Office.Clients
  • Vlan3: Operation IT (some IoT Devices) --> should have access from Identity SideA.IoT.Devices

For now everyone would have, per the guides available, one single Identity/attribute. That would be a devastating approach to network security. Everyone who has access to the IP of the router would be able to use this Identity.

This is even a bigger problem if we talk about Cloud or Multi Cloud environments!

YES, you could limit the access to the Router Host by Firewall installed at the location or cloud environment. Which doubles the amount of devices, efforts to maintain and cost of ownership per location.
YES, the Host has its own firewall (ufw) to not double costs.

BUT: You are still limited to use one Identity for IoT and User Devices which, again, would be a network security problem.

Possible Solutions and drawbacks:

  1. Create "Top-Rack" routers for every VLAN. Every router can have their own Identity and can be located across networks to be only reachable from the destined subnet either by central firewall or host firewall.
    Positive: Every LAN will have their own identity.
    Drawback: every LAN needs their own router either virtual or hardware provided which will skyrocket costs and maintenance.

  2. Use a zitified Caddy reverse proxy with the client_ip filter (not tested yet). Hypothetically, you would be able to install/bind your Identities to a zitified Caddy reverse proxy. The idea is to use something like the following. I do not know if multiple bindings are working and if the caddy file is correct, but that's what I'm currently investigating to solve this problem.
    Positive: Based on the Match of the client IP, the right identity is chosen and routed through the overlay network. This can be achieved, if caddy reverse-proxy can handle multiple identities, via one instance only. would separate the ZeroTrust-network from the "source-desider"
    Drawback: Second maintenance and upgrade costs at every router level and different management. The larger the network, the larger the maintenance effort. If the Host is corrupted, the Identities could be modified, which would create a whole in the authentication process.

Idea of Caddy file:

{
admin off
http_port 8080
}
localhost:8080
@Vnet1 {
client_ip 192.168.1.0/25
expression Vnet1('192.168.0/25')
}
@Vnet2 {
client_ip 192.168.128/25
expression Vnet2('192.168.128/25')
}
reverse_proxy @Vlan1 {
# intercept address, will be used if service id not set in the 'ziti' block
transport ziti {
# required, ziti identity file
identity {$ZITI_IDENTITY_Vnet1}
}
}
reverse_proxy @Vlan2 {
# intercept address, will be used if service id not set in the 'ziti' block
transport ziti {
# required, ziti identity file
identity {$ZITI_IDENTITY_Vnet2}
}
}
@denied not client_ip Vnet1 Vnet2
abort @denied

  1. Ziti Option:

Favorite solution:

A matcher-config within the intercept at OpenZiti Level. The Intercept is only hit/recognized if the source IP matches the network configured. This would keep things clean and simple from the start. So I can create, let's say, transit services.

Let me know what you are thinking about that and if there are already solutions or roadmaps planned to that.

Thanks all!

That's great feedback and I hear it. OpenZiti does take someone with specialized knowledge but I don't know if it's any more than the knowledge of any other network/security engineer doing the same sort of thing with other techn like wireguard, tailscale, cloudflare, ${insert other tech}. That's one of the reasons CloudZiti exists, it eliminates all those needs. Once you get over that initial hump though, it should be quite smooth sailing. We keep trying to lower that hump and make it as easy as possible to get over it...

I would challenge this point, though I fully admit that my domain expertise clouds my "new learner" vision. Still, in my opinion, OpenZiti has a very flexible architecture and allows people the freedom to start small and slowly, and ramp up. That's the reason OpenZiti supports ZTNA deployments and ZTHA deployments. To allow people to start with what they're more familiar with and grow into zero trust. You should not need a "big bang style" deployment.

Assuming I understand your described layout, you're correct. This would be a "ZTNA" approach. It is most definitely the "least" of the zero-trust approaches for the reasons you have outlined. If you are accepting traffic at a router on your LAN, that traffic can source from any, untrusted machine. You're placing a LOT of trust on your LAN security in this design. While this is a common practice, it's certainly not a "best practice" when it comes to zero trust but this is the design of numerous networks.

I'm going to skip past the possible solutions 1 and 2 and go to the 'favorite' solution because it's based on an IP address. IP addresses are not strong identities and we can't/don't use them for authentication. If this is an area of innovation that you'd like to explore, it sounds like a fun project for someone but I don't think it will match the OpenZiti motif.

Instead, the better but "more work" approach is to deploy tunneler apps on each and every machine that needs access. Then you will have a strong identity on every machine that needs to participate on the network and you have moved over to ZTHA (not ZTNA). ZTHA is a stronger level of zero trust than ZTNA, undoubtedly, but it's still allows any local application on that machine to send traffic. If a local app gets compromised, it would be able to attack any service that identity has access to. Generally, that's an acceptable security posture for most organizations/people. The last step on the journey, ZTAA (application embedded zero trust) eliminates even this worry, but for many people it's a longer step to get there. Some people can start there! (lucky them) :slight_smile:

One other popular alternative to IP which is still a strong identity is to use mTLS. With mTLS, the challenge is similar (if not identitcal) to the OpenZiti challenge of deploying identities everywhere, you need to get that strong identity (x509 cert) deployed to the clients that plan to connect to the zero trust proxy. In my opinion, this is nearly identical to the challenge of deploying and enrolling OpenZiti tunnelers but it's another approach that would at least utilize a strong, cryptographically identifiable identity.

Hope that helps. I'm happy to see what anyone else in the community thinks!

1 Like

I would take a different approach. You can crawl, walk, run to Zero Trust with OpenZiti. Zero Trust architecture, like any security architecture, is about risk reduction. If an organization knows where its major risks live, then those are the initial targets. Database server housing critical company information? How many people need direct access? Apply OpenZiti and protect that asset, then move on to the next down the list. This will give you the most risk reduction, with the least effort, allowing advanced engineers the opportunity to develop their skills, figure out how to integrate it into the organization, and generally plan a transfer of responsibilities to the network operations and other teams and execute that plan over time.

On the ZTNA point, there is a feature that can help make the gateway model a little more secure while you are deploying more clients and moving to a full implementation. While I agree using IP addresses is not the best path, we do collect detailed data on the incoming sockets. The fabric.circuit created events will include local initiator IP addresses and ports. This is the socket that is dialing the service. Combining that with local logs can give an option while full deployment is pending. It is not a cryptographic identity, but when paired with DHCP logs, etc., it can be very helpful. These records also allow you to monitor usage of various sockets on the other end, as they are collected for egress gateway models as well. Mapping routed subnets via OpenZiti would keep the routing as it is today, and parsing the records could then do a basic service discovery model, tell you what IPs dialed what IPs. Parsing that with subnet maps and more detailed logs if necessary can give you a very good starting place for service design when you move to a more complete implementation.

1 Like

Hey Clint,

thanks for the reply, and I want to underline that I totally have the same view point on this.
To have a trust only on router level would not really benefit the idea of ZeroTrust and the OpenZiti project.

From a client perspective, most of them having problem with the adoption cycle. Most of the companies in Europe have different managed Service providers, which needs to be carefully managed and replaced by adopting to this technology.
In my special case, the Implementation of the "WAN" will start earlier than the possibility to deploy Zero Trust Tunneler Clients on every User. They are also not able to wait for the Client option to become available, as their "WAN" contract is ending. So I will have a time window where every location needs to be addressed by ZitiRouters as Lan-Gateways.
Only during this window of managed service provider changes, we need to implement it in such a way. We also can't start with a "legacy" network project. This time window is too short to purchase and operate 350 WAN routers and switch afterward to a full-fledged OpenZiti implementation. As any company, cost is always a problem and this kind of "double-project" would be acceptable due to the stranded costs.

The End-game will definitely be a real ZeroTrust Network with direct communication between Host and App Identities.

These kinds of timing-problems are affecting a lot of my clients who want to get their Company into ZeroTrust.

Thank you for your comments and motif, as I said I totally agree with your statements.

Hey Mike,

thanks for your comments! I really appreciate the community feedback in these sections!

As I already wrote in my answer to Clint It's not about the technical adopting, it's more a timing problem. Some Contracts for Technologies are ending and some are starting at different times. That's what I meant with the "Big-Project" in my first post.
The asset staged approach is perfect, and this is what we cover in cases of Applications or smaller footprints. If you are talking about a whole company network, Locations and Services needs to be included by day one.
But you gave a good hint what it means to work in a large quantity, thanks for that!

Thanks for the technical input about the Lan-gateway-secure approach! I will take a look into this!

As in Clints answer, I also share your opinions about Zero Trust.

Hello,

One thought on your Ziti Option (3) that drops connections from unwanted clients based on where they're coming from (source IP) is to configure the service to:

  1. pass the intercepted client's source ip from the intercepting tunneler/ER to the hosting tunneler/ER by specifying "sourceIp": "$src_ip:$src_port" in your service intercept.v1 configurations, and
  2. white list the allowed client IPs by specifying them in the allowedSourceAddresses field of the host.v1 configurations of your OpenZiti services.

One obvious downside of whitelisting at the hosting tunneler/ER is that connections would be intercepted and proxied over the OpenZiti network before ultimately being dropped, but I think it otherwise comes pretty close to what you're looking for, and btw it is a capability that OpenZiti has today.

Mainly I'm putting this out there to see if getting close to what you have in mind. If so, an improvement could be to implement an allowedSourceAddresses field in the intercept configuration, so unwanted clients are immediately dropped at the gateway. I don't think this would be very difficult, btw.

1 Like

Hey Shawn,

Thanks for your reply!

I tested the approach in my test environment. I was not 100% sure how to configure the source IP stats at first hand, works like you described. Thanks for the input.

Like you described, both intercepts (wanted and unwanted traffic) are moved to the other edge and only the wanted one (matching source IP) is released.

Is there any chance to get this configured not just for single IPs but for networks (e.g. 192.168.1.0/24)?

Hey Frederik,

Wow, that was quick work! Thanks for trying out the approach so quickly. Did you try putting CIDR values in the AllowedSourceAddresses list? It should work.

A little background:

If you're using an edge router at the terminating endpoint, which I'm guessing you are, the ER adds any allowed source addresses to the loopback interface. When the router connects to the underlay address in the service configuration, it binds the socket to the source address that the intercepting tunneler projected via the sourceIp field. If the address doesn't exist on the system, the bind fails and the connection is dropped. So any address that can be added to a network interface can be specified the AllowedSourceAddresses field.

So as I mentioned previously, this approach of filtering incoming connections at the hosting endpoint works in a proof-of-concept capacity but is not ideal. For one thing it allows doomed connections to occupy resources on the gateway. It also involves managing the source IPs on the target hosts. The AllowedSourceAddresses field in host.v1 really exists to control which source IPs can be projected from intercepting tunnelers. Adding AllowedSourceAddresses to the intercept.v1 configuration would remove both of these downsides, and I think we should talk about that if you're ready to build out an OpenZiti network with source IP filtering.

1 Like

Thanks for the explanation! Good to know how the Technology behind this is coming to play.

Seems I mixed something up here.

I tried putting in CIDR, as this did not work, I do get now a timeout on the local loop back.
The Settings and configs are visible in the log.

Just for reference, my Test Setup (Azure):

Vnet4(LAN, 192.198.2.0/24) Client1: 192.168.2.4
<--peering-->
Vnet3(Router Vnet, 10.0.2.0/24) Router2 10.0.2.4
<--OpenZiti-->
Vnet2(Router Vnet, 10.0.1.0/24) Router1 10.0.1.4
<--peering-->
Vnet1(Server, 192.168.1.0/24) Server1: 192.198.1.4

All Routes, Peering etc are configured correctly. Without the src_ip and the allowed src IP is working.

My OpenZiti network does also host a Windows Tunnler(PC), Android Tunnler(Smartphone) and a Zitified Service(Python http server)

My Settings are:
Config - Host.v1 (Yes the web.az.test refers to a azure private DNS Zone which is solved correctly to 192.168.1.4, that's awesome btw - solves a lot of problems)

Config - Intercept.v1

In the Log I can see that the Source IP passed but:

"error creating route for [c/ZSo5aTIT-]: dial tcp 192.168.2.4:0-\u003e192.168.1.4:8000: i/o timeout"

If I try to access it from my Windows PC etc it cant bind the address like you described:

dial tcp 100.64.0.1:0-\u003e192.168.1.4:8000: bind: cannot assign requested address"

Your configurations look right to me. I should say that the AllowedSourceAddresses field will only be correctly used by routers that are running on Linux, and it should not matter which OS the initiating tunneler is running on.

  • the timeout error is very odd. I'm not sure how binding a source address to the socket could cause that. I'll take a closer look at the ER for potential issues. Are you sure that the server at 192.168.1.4:8000 was running (and routable from the ER host) when this timeout happened?
  • the source address of 100.64.0.1 makes sense if your tunneler on Windows was intercepting a local connection, because that's the default IP of the tun interface that is used for intercepting packets.

fwiw you should be able to see the allowed source addresses attached to the "lo" interface (ip addr show dev lo) on the host that's running the hosting ER once the ER receives its service configurations.

So there may or may not be some unexpected issues with the source IP filtering at the hosting tunneler/ER. I'll look into those separately, but I do think your best option is to use source filtering at the intercepting tunnelers, once we implement it of course :smiley:

Hey Frederik,

I tried the configuration that you described here, and the setup worked for me. In the process of setting it up I had an idea about why it might not be working for you.

I'm guessing that your end server (the one at 192.168.1.4:8000) is not running on the same host as the ER? If so, the timeout that you see is likely caused by there being no route on your server host that directs the source IPs to the ER host. This route would be necessary for the return path for connections that are initiated from the ER to the server host.

If this is what's going on, you could work around it by adding a route for the source IPs on the server host so the packets have a return path. e.g.:

   pweb-host$ sudo ip route add 192.168.2.0/24 <lan_ip_of_host_running_er>

Again, all of this is just to get the basic functionality working for you.

A more direct approach to filtering intercepted connections by source IP is to add an allowedSourceAddresses white list to the intercept configuration, so there's no need to project a source IP over to the remote edge router. We need to flesh out exactly how this whitelist would be implemented. We'll stay in touch on that.

1 Like

Hey Frederik,

FYI I made a first pass at adding source ip white listing at the intercepting ER. If you want to try it out you can get the ziti binary from the github workflow run. Run your local edge router using this binary.

You can also use this binary to run your ziti controller. If you do, the intercept.v1 config type schema will automatically be updated to include the new allowedSourceAddress field when you first start the new controller. If you'd rather leave your controller alone for now, you could alternatively update the intercept.v1 configuration type in your existing controller with the ziti cli:

curl -LO https://raw.githubusercontent.com/openziti/edge/intercept.srcip.filter/tunnel/entities/intercept.v1.json
ziti edge update config-type intercept.v1 -f intercept.v1.json

The config type schema will now have an allowedSourceAddresses field which can be populated with ips and/or CIDRs. Any entries in the list are passed as the src ip parameter when creating the iptables tproxy rules.

Let me know how it goes if you give it a shot, or if you have any questions.

2 Likes

Hey Shawn,

wow! Thanks for the work and the fast adoption!
I'll give it a try as soon as possible!
I'll let you know as soon I implemented it.

Best Regards,
Frederik

1 Like