Thank Clint for the great tips 
, I will give them a go and see how they work.
PhilipGriffiths has been in touch with me regarding how our experience with Ziti has been going, so am just adding a follow-up of our current experience with Ziti
During all of our tests, Ziti tunneler has worked seamlessly and allowed us to access our iot devices in a reliable manner, however we have only been able to define and access one site at a time which is a big limitation.
The biggest issue encountered by us in being able to use the Ziti network overlay as a replacement for Teamviewer VPN is that it is currently not possible to define a single Ziti service that is provided by multiple endpoints. Based then on who we need to connect to, tell the tunneler to direct the traffic to the selected endpoint.
Background:
The iot devices we manage are industrial devices which only allow access to them via an IP address. For convenience of managing projects with identical devices, they all have the same subnet/IPs.
All the Industrial Iot hardware is reachable only via IP address
The number of sites we have are in the 1000’s with devices in the 10000’s and we require to be able to easily administer access to sites. There are approx. 10 admin PCs.
Conceptually we should be able to define a single “Service” (Reachable IPrange/ports e.g. 192.168.1.0/24:102) and assign this single service to each site. All that will then be needed is to give each admin PC Dial permission to the service and they will automatically be able to access all Sites and corresponding Iot devices. This will avoid having to define a service for each site and individually give each admin access to each service.
Our Current mode of operation :
- Open a VPN using an ID of the remote Windows PC connected on the “Site’s Iot network”
-
Add static rule on Admin PC side to be able address the remote IP address via remote PC
Route add 192.168.1.0/24 7.192.42.17 -
Devices can be reached directly from admin PC
e.g. ping 192.168.1.200 will work
Our current requirements would be to
- Be able to address from Admin PC our Iot devices using the remote IPaddresses eg 192.168.1.200
- Easily manage access permissions to 1000s of endpoints, adding or removing Admin PCs
As a sidenote, while looking through the tunneler code I came across
Intercept Address Matching
Tunneler SDK versions 0.18.16 and earlier would associate incoming packets with the first service that had a matching intercept address to the packet’s destination proto:ip:port. Now the tSDK associates packets with the ziti service that has the most precisely matching intercept address. “precision” is defined as follows:
- ip addresses with higher prefixes are more precise than those with lower prefixes
- exact hostnames are more precise than wildcard domains
- smaller port ranges are more precise than larger port ranges
- an ip address or hostname match of any precision is more significant than a port range match
………
Not sure if something similar using the site Domain name of the remote site as additional parameter could be a worker.
If I understand your requirement correctly, we don’t have the exact feature you are looking for. We have a feature called addressable terminators though, where one service allows to reach multiple endpoints using the intercepted ip address. In this case, an admin pc would be able to reach each endpoint by using the ip of that endpoint. You need to run ZET on each IoT device, and that just is not possible in your case. Thus, you would run ZET on an IoT gateway and the existing feature is limited to one IP per ZET and you need is to have ability to address multiple IPs per ZET i.e. subnet per ZET. The code is extendable to perhaps enable something like it, but it requires development time.
Here is how the enhancement to addressable terminators might look like to me at least at this moment. It could change in the final solution though.
- Destination NAT would need to happen as well to take care of the ip subnet overlapping.
- Due to NAT in 1, pc admin would need to address each endpoint by a unique ip address, no overlap across region, even though under the hood the NAT translation would happen to real IP addresses. i.e.
ssh admin@10.10.0.10 (site #1 endpoint # 10) but real ip address 192.168.0.10 ssh admin@10.10.1.10 (site #2 endpoint # 10) but real ip address 192.168.0.10, etc
It sounds like this is the only showstopper for you. Thus, if we were to address this requirement , then you would replace teamviewer with openziti?
Thank you for keeping OpenZiti in mind for this long! This is a very involved threads with a lot of interesting ideas in it. Every time I start thinking on how we could make this work with tunnelers and some combination of cidr and/or wildcard dns intercept and addressable terminators, I skim back through the thread and I find a nugget of info that knocks my idea down!
The main “nuggets” that keep knocking our ideas down are:
- The software that initiates connections is proprietary, and can only be configured with IPs (no hostnames).
This rules out any clever use of dns intercept addresses (wildcard or not). - You’d like to always connect to a 192.168.1 address, and select the site before initiating connection.
I think this rules out any kind of NAT between the admin and remote PCs. - The network is the same at all sites - 192.168.1/24.
This limits us to a single service. - A tunneler only has the intercepted protocol:ip:port to go on when figuring out which service (and terminator) to
dial.
Ignoring (4) for a moment, we could use a subnet service that intercepts 192.168.1/24 with address/port forwarding enabled in the host configuration. Of course we need a way to select the site that we want to connect with - this is where addressable terminators are tempting… But as mentioned in (4), tunnelers can only use the destination ip, etc. to determine the terminator identity (via “$dst_ip”, et. all in "dialOptions":{"identity"}.
So with all of this in mind I think one way to do this with OpenZiti tunnelers could be to support new variable references in "dialOptions":{"identity"} that resolve to values that can be externally influenced. for example: “$file:/path/to/file/containing/sitename”. I’m not saying this is great or that I’ve even thought it through completely, but it could work.
edit:
To be clear, I don’t think this is an approach that we would want to support in the released tunnelers. The potential for surprises from a security and an OS-dependency standpoint don’t seem worth the effort and risk of folding such a feature into the main products.
We normally think of solutions that use the OpenZiti tunneler in a way that would allow access to any endpoint (site in your case) at any time, without pre-selecting the site before making a connection. I think this is why this thread has been running for so long. We’ve been offering solutions that would allow (and require) you to address the site when using the client software. I think this approach is different from your goal of selecting one site at a time with a (VPN or other) connection.
Assuming I’m understanding your current (and desired) workflow correctly:
- select the target site for the next connection (currently by VPNing to the site and updating a static route)
- initiate connection with client software, presumably using the same configuration which points to 192.168.1/24 addresses
If you want to continue with this workflow and basically replace the TeamViewer VPN with an OpenZiti tunneler then I think one of @TheLumberjack’s earlier suggestions of creating an OpenZiti identity for each site and restarting the tunneler with that identity would be a decent option. We would need to make some relatively minor and supportable changes to the tunnelers, but they would be changes that we’d be comfortable standing behind.
Another option that would fit well (even perfectly) with this workflow would be to write your own OpenZiti app based on the tunneler SDK. You could even wrap a nice UI around it and have a searchable pulldown for your site selector. Of course this is more work than running a tunneler.
Thanks so much for all the suggestions
Destination NAT isn't really what we require as also stated by scareything
Agreed, whatever is implemented has to have no surprises in the future and be completely secure.
Using Lumberjacks suggestion of creating an OpenZiti identity for each site and restarting the tunneler with that identity might not be so ideal since we are talking of managing 1000's of sites multiplied by the number of admins which may change over time.
The ideal solution in order to replace any sort of VPN would be a "Gateway app" based on the tunneler SDK that exposes only the gateway IP to the Ziti Network permitting routing of traffic to any of its connected local networks.
This would achieve the objective of having a zero trust Ziti cluster of reachable devices.
any other ideas/suggestions are appreciated
thanks again
@Danieleb Ok, what about this. I think it is pretty close to what you need, but it does not come with a pretty UI though. You can use an existing addressable terminator service. I have tested this on my win 10 client and edge routers with tunneler option. But I don’t see any issue running this set up with ZET instead of ERwT.
Basically, this solution is using the ssh tunneling to accomplish what you do and gives you ability to access more sites than one without switching and opening a path for the entire network across your VPN, only selected ports. Unless the number of ports per devices is big too, then it may not make sense.
Here are the details for the service config. It is pretty straight forward and easy. The wild card dns name used is just an example, and you can pretty much use whatever you would like.
IotAddressableService-intercept.v1
{
"addresses": [
"*.iot.service"
],
"protocols": [
"tcp"
],
"portRanges": [
{
"low": 22,
"high": 22
}
],
"dialOptions": {
"identity": "$dst_hostname"
}
}
IotAddressableService-host.v1
{
"port": 22,
"address": "127.0.0.1",
"protocol": "tcp",
"listenOptions": {
"identity": "$tunneler_id.appData[fqdn]"
}
}
I tested this with app data to configure the addressable identity name, but in your case this probably makes more sense "identity": "$tunneler_id.name".
Once this service is configured and all the policies are in place, then you would control the traffic flow with ssh -J and ssh -L. it is easily scriptable too.
ssh -fNn -L2122:10.10.1.166:22 -L13389:10.10.1.6:3389 ziggy@ashburn.iot.service
ssh -fNn -L2022:10.10.1.166:22 -L23389:10.10.1.6:3389 ziggy@phoenix.iot.service
You just need to use different local ports. Obviously you are limited to the number of ports. If you are using ssh to access these devices, then you just jump to them using -J option.
ssh -J ziggy@ashburn.iot.service ubuntu@10.10.1.166
ssh -J ziggy@phoenix.iot.service ubuntu@10.10.1.166
dariuszSki thanks so much for your input and sticking with this problem, much appreciated…
I have been considering this option since yesterday, the solution seems interesting however I see some problems which may not be addressable.
-
The proprietary software we use does not allow any other IP in the request apart from the actual IP of the device so not sure we would be able to route the 192.168.1.0/24 traffic over SSH
-
The proprietary software we use does not allow any other PORT in the request apart from the actual PORT of the device ie 102
Apart from this it could be a worker as long as at the preselected “SiteX” we are able to route the traffic to the devices.
I have made a quick diagram of how it would have to work in this case
thanks
Would you allow your admin's to modify the overlay? Meaning, would you let them run a command ziti CLI command to change the overlay? Back in May, the idea that I thought would be easiest, but would require you to allow your 'admins' to change the overlay, was to give every admin their own identity and then just let them change it but I think an easier approach is to allow every administrator to have their own service that they can change. It's effectively the same as toggling the VPN connection like your image and works like this...
- declare one "host.v1" configuration, that covers all remote sites, and uses the addressable terminator functionality.
- provision
ziti-edge-tunneldevices on all your sites, giving the identity for each site an attribute. Something like "zero-trust-bastion" - for every 'administrator', you make a single identity for them AND you declare a new intercept config for them and a service for them
When New Sites Come Online
When a new site comes online, you would provision a new identity with the "zero-trust-bastion" attribute, and deploy/enroll a new ziti-edge-tunnel at the remote site in the 192.168.1.0 address space
When New Admins Come
When a new admin comes, you would need to:
- provision an identity for the new admin
- provision an intercept config for the new admin
- provision a new service for the new admin
- grant / give them the ability to modify the overlay
Accessing 'n' remote sites
When "admin1" wants to access "site1" - the admin would need to run a script (or a raw ziti cli command) that basically boils down to changing the addressable terminator for their identity. it would look like this:
Change Admin1 to use Site1
ziti edge update config 'admin1.config.intercept.v1' -d '{"dialOptions": {"identity":"site1"}, "protocols":["tcp"],"addresses":["172.31.50.0/24"], "portRanges":[{"low":80, "high":80}]}'
Change Admin1 to use Site2
ziti edge update config 'admin1.config.intercept.v1' -d '{"dialOptions": {"identity":"site2"}, "protocols":["tcp"],"addresses":["172.31.50.0/24"], "portRanges":[{"low":80, "high":80}]}'
The Caveat
The caveat here is those remote admins need to have the power to change the overlay network. Maybe that's ok? I'd also probably write a tidier script for them to run to make it easier on them.
I'm going to demonstrate this live on Ziti TV in 30 minutes, if you're interested
Yes, this is very good solution given the restrictions. What would it take and does it make sense to add the ability to change dialOptions in the ZDE GUI, like we can do with DNS IP Range?
Nevermind it is a service based change not local ZET change, or perhaps sdk like ZET can pass dynamically using the appData key pair with the identity I would want to talk from now on.
i.e.
“dialOptions”: {
“identity”: “$appData[key_name]”
}
Link to the Ziti TV where we start talking about this Ziti TV Oct 21 2022 - Config Types take 2 - YouTube
If interested here are the commands which I ran prior to today’s Ziti TV:
ziti edge create config 'admin1.config.intercept.v1' intercept.v1 '{"dialOptions": {"identity":"site1"}, "protocols":["tcp"],"addresses":["172.31.50.50"], "portRanges":[{"low":80, "high":80}]}'
ziti edge create config 'remote.access.config.host.v1' host.v1 '{"forwardProtocol":true, "allowedProtocols":["udp", "tcp"], "forwardAddress":true, "allowedAddresses":["172.31.50.0/24"], "forwardPort": true, "allowedPortRanges": [ { "low": 1, "high": 65535 }], "listenOptions": {"bindUsingEdgeIdentity":true}}'
ziti edge create service 'remote.access' --configs "admin1.config.intercept.v1","remote.access.config.host.v1" -a "admin.accessable.services"
ziti edge create service-policy admin.access.bind Bind --service-roles "#admin.accessable.services" --identity-roles "#zero-trust-bastion"
ziti edge create service-policy admin.access.dial Dial --service-roles "#admin.accessable.services" --identity-roles "#remote.administrators"
## i also created/enrolled three identities
ziti edge create identity device site1 -a "zero-trust-bastion" -o site1.jwt
ziti edge create identity device site2 -a "zero-trust-bastion" -o site2.jwt
ziti edge create identity device site3 -a "zero-trust-bastion" -o site3.jwt
Contents of changeSite.sh
ziti edge update config 'admin1.config.intercept.v1' -d '{"dialOptions": {"identity":"'"$1"'"}, "protocols":["tcp"],"addresses":["172.31.50.0/24"], "portRanges":[{"low":80, "high":80}]}'
Thanks Clint, I will try set this up my side and give a try to see if this solution can be workable for us. Thinking about it, I don’t think it should be much of an issue to let our admins change the overlay, but I first need to test it out properly and get back to you.
2 Likes
Even though I am not too sure regarding the fact that all admins edit the overlay as it essentially overrides the concept of “Zero trust”,
I have been giving this solution a shot and have most likely misunderstood/mistaken something in terms of configuration and am not sure what. Using just one admin device all works as expected and am able to switch sites.
On adding a second admin, each with access to different sites, I end up with both admins accessing the same site.
Clint I’d appreciate it if you could give me a pointer in the right direction. Here are the exact commands I have run on my setup.
#Intercepts for Admins
ziti edge create config ‘admin1.config.intercept.v1’ intercept.v1 ‘{“dialOptions”: {“identity”:“site1”}, “protocols”:[“tcp”],“addresses”:[“192.168.1.0/24”], “portRanges”:[{ “low”: 5938, “high”: 5938}]}’
ziti edge create config ‘admin2.config.intercept.v1’ intercept.v1 ‘{“dialOptions”: {“identity”:“site2”}, “protocols”:[“tcp”],“addresses”:[“192.168.1.0/24”], “portRanges”:[{ “low”: 5938, “high”: 5938}]}’
#Host config
ziti edge create config ‘remote.access.config.host.v1’ host.v1 ‘{“forwardProtocol”:true, “allowedProtocols”:[“udp”, “tcp”], “forwardAddress”:true, “allowedAddresses”:[“192.168.1.0/24”], “forwardPort”: true, “allowedPortRanges”: [{ “low”: 3128, “high”: 3128 }], “listenOptions”: {“bindUsingEdgeIdentity”:true}}’
#Services for Admins
ziti edge create service ‘remote.access’ --configs “admin1.config.intercept.v1”,“remote.access.config.host.v1” -a “admin.accessable.services”
ziti edge create service ‘remote.access2’ --configs “admin2.config.intercept.v1”,“remote.access.config.host.v1” -a “admin.accessable.services”
#Bind and dial service-roles
ziti edge create service-policy admin.access.bind Bind --service-roles “#admin.accessable.services” --identity-roles “#zero-trust-bastion”
ziti edge create service-policy admin.access.dial Dial --service-roles “#admin.accessable.services” --identity-roles “#remote.administrators”
Site identities
ziti edge create identity device site1 -a “zero-trust-bastion” -o site1.jwt
ziti edge create identity device site2 -a “zero-trust-bastion” -o site2.jwt
ziti edge create identity device site3 -a “zero-trust-bastion” -o site3.jwt
#Admin identities
ziti edge create identity device admin1 -a “remote.administrators” -o admin1.jwt
ziti edge create identity device admin2 -a “remote.administrators” -o admin2.jwt
Contents of changeSite.sh
ziti edge update config ‘admin1.config.intercept.v1’ -d ‘{“dialOptions”: {“identity”:"’“$1”‘"}, “protocols”:[“tcp”],“addresses”:[“192.168.1.0/24”], “portRanges”:[{ “low”: 5938, “high”: 5938}]}’
thanks
Glad to see you’re giving it some thought. I think what I’ll do is make one big huge follow-up where I go back through this thread and sum up all the options we have. So far - not a single one of them is “perfect” for your use case…
It’ll take me a while - but I’ll try to get to that today
Oops I see I can answer this one though. I'll take a crack at that now
I think this is the closest solution by far. I just need to think about how I can set it up that each admin can only switch the overlay for the “site identity” associated with their “admin identity” and that would make it ideal.
I am sure @TheLumberjack will have better answer than mine. As far as I can see this is a service level change. You want a client level change and that is not available at the moment for your use case. Perhaps, the developers may have some tricks up their sleeves to help you with that.
Ok, I’ve gotten back to take a look at this… I think I was wrong with the service-policy example I gave. I stated before that every administrator needs their own:
- identity
- intercept config
- service
But then in the example commands, I didn’t show that, instead I showed only a single service. So what’s happening here is the service-policy grants “all admins” access to “all the services” (of which there is only one service and where I went wrong). In reality, you want one service/service-policy per admin too. That way only “admin1” can access “service1”. Only “admin2” can access “service2”. etc.
I think that’ll fix the issue.
admin2
#Commands for Admin1
user=admin1
ziti edge create config "${user}"'.config.intercept.v1' \
intercept.v1 '{"dialOptions": {"identity":"site1"}, "protocols":["tcp"],"addresses":["172.31.50.50"], "portRanges":[{"low":80, "high":80}]}'
ziti edge create service "remote.access.${user}" \
--configs "${user}.config.intercept.v1","remote.access.config.host.v1" \
-a "${user}.accessable.services"
ziti edge create service-policy "${user}.access.dial" Dial \
--service-roles "#admin1.accessable.services" \
--identity-roles "#zero-trust-bastion"
admin2
#Commands for Admin2
user=admin2
ziti edge create config "${user}"'.config.intercept.v1' \
intercept.v1 '{"dialOptions": {"identity":"site1"}, "protocols":["tcp"],"addresses":["172.31.50.50"], "portRanges":[{"low":80, "high":80}]}'
ziti edge create service "remote.access.${user}" \
--configs "${user}.config.intercept.v1","remote.access.config.host.v1" \
-a "${user}.accessable.services"
ziti edge create service-policy "${user}.access.dial" Dial \
--service-roles "#admin1.accessable.services" \
--identity-roles "#zero-trust-bastion"
Another option that you might consider, if you’re inclined, might be to build a custom service that automates the control plane of your Ziti network. Rather than allowing admins to directly interact with the network control plane, you could build a service with its own authentication mechanism, that allows admins to interact with the control plane of your Ziti network in a limited, controlled manner.
The rules and semantics around your configuration would be baked into that service.
We’re exploring these kinds of approaches internally, and you might be seeing some new stuff from us fairly soon that illustrates these ideas in action in a number of interesting application scenarios.
Something is still not right with the configuration logic.
Starting off with only the following identites pre configured: admin1,admin2,site1,site2
When I run the following scripts
ziti edge create config ‘remote.access.config.host.v1’ host.v1 ‘{“forwardProtocol”:true, “allowedProtocols”:[“udp”, “tcp”], “forwardAddress”:true, “allowedAddresses”:[“192.168.1.0/24”], “forwardPort”: true, “allowedPortRanges”: [{ “low”: 3128, “high”: 3128 }], “listenOptions”: {“bindUsingEdgeIdentity”:true}}’
then
AddAdmin.sh admin1
user=$1
ziti edge create config “${user}”‘.config.intercept.v1’
intercept.v1 ‘{“dialOptions”: {“identity”:“site1”}, “protocols”:[“tcp”],“addresses”:[“192.168.1.0/24”], “portRanges”:[{ “low”: 5938, “high”: 5938}]}’
ziti edge create service “remote.access.${user}”
–configs “${user}.config.intercept.v1”,“remote.access.config.host.v1”
-a “${user}.accessable.services”
ziti edge create service-policy “${user}.access.bind” Bind
–service-roles “#${user}.accessable.services”
–identity-roles “#zero-trust-bastion”
ziti edge create service-policy “${user}.access.dial” Dial
–service-roles “#${user}.accessable.services”
–identity-roles “#remote.administrators”
Don’t think it is right as I get following output from
Output format: STATUS: ID (ID ROUTERS) → SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
OKAY : site2 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : site1 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : admin2 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : admin1 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: Y Bind: N
admin2 somehow gets dial permissions dor admin1 services
adding the second admin configs the same happens to admin 1
Output format: STATUS: ID (ID ROUTERS) → SVC (SVC ROUTERS) Common Routers: (ONLINE COMMON ROUTERS/COMMON ROUTERS) Dial: DIAL_OK Bind: BIND_OK. ERROR_LIST
OKAY : site2 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : site2 (1) → remote.access.admin2 (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : site1 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : site1 (1) → remote.access.admin2 (1) Common Routers: (1/1) Dial: N Bind: Y
OKAY : admin2 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : admin2 (1) → remote.access.admin2 (1) Common Routers: (1/1) Dial: Y Bind: N
ERROR: ziti2-edge-router
- Identity does not have access to any services. Adjust service policies.
ERROR: Default Admin
- Identity does not have access to any services. Adjust service policies.
OKAY : admin1 (1) → remote.access.admin1 (1) Common Routers: (1/1) Dial: Y Bind: N
OKAY : admin1 (1) → remote.access.admin2 (1) Common Routers: (1/1) Dial: Y Bind: N