I think I want this to be an “Intercept.v1” config, so that it intercepts on the ziti client tunneller software, and then outputs on a remote network that is not reachable from the Internet.
If I add this as a host.v1 config, then it does not show up in the client tunneller software as a service. Is it right to assume that I need an “intercept” config, and then I require a corresponding “terminator” to make that work?
I would like to intercept “192.168.14.0/24” on the “Ziti Desktop Edge” software and then route traffic to that remote subnet that shares an interface with a remote Ziti router
generally speaking i would suspect that your remote tunneller is using an identity which does not have a “Bind” policy assigned to it. Making a “Bind” policy will allow the tunneller to “host” that service (meaning serve as a termination/exit point)
Here’s a bunch of commands I just whipped up… I think this is what you need. You probably have most of this already but if not - here you go… i’d also suggest not opening ‘every port’ but - you’ll figure that part out
I have tried to setup a network according to the commands given by you in this post and it works great.
We are however trying to do something slightly different and I am not sure if it is possible and how we would go about it.
In brief we have many remote IOT devices which are not “Zitifiable” and which all have the same local IP / subnet (eg. 192.168.1.5/24). We are able to setup a separate endpoint (eg. 192.168.1.1/24) at each of these sites with an onboard “ziti-edge-tunnel” or Custom-SDK application.
However as we understand it, with the given ZIti setup we would need to define both a service and an endpoint for each of these locations. Our “administration” endpoints would then need to be setup to have access to each and every service, which becomes very cumbersome and is definitely not scalable.
Is there any easy way of setting up hundreds of endpoints each exposing the same “subnet services”, which are then reachable only by “Administration endpoints” when needed?
ie Identity “Admin” is able to connect (only when required) to Identity “IOT_GW_100” which exposes subnet 192.168.1.0/24 and gives access to IOT_Device_100 (192.168.1.5/24)
This is going to be a long response because you’ve asked an excellent question. But as with most things - there’s a lot of nuance in this response so hopefully this big wall of text doesn’t scare you off…
At the bottom of the post is a diagram that I made. If it’s wrong please correct it. This is my understanding of what you’re looking to do ‘roughly’.
Custom SDK App?
The first big question from me is about what is connecting to these sites on the ‘left side’ shown. Are you expecting to author code and make a custom ziti app that knows how to do these things? I expect “no” but thought I’d ask… Same is true for the far side. When you’re writing code on both sides - well… you can do ANYTHING you want!!! and you could absolutely do what you want to do.
I expect that’s writing code is not what you want, since that’s “more effort” (and hey, I don’t blame you ). With a tunneller on the left side your options do start to narrow since you’ll be reliant on the tech we’ve implemented to date. Assuming you’re using a tunneller on the left, at this time, I don’t see how you can really get away from declaring “a bunch of services”, a block of them for each site. Yah, I agree it’s more cumbersome but with APIs/ziti cli it can all be scripted out. It might not be “that bad” but I do understand the point you’re making - certainly not ideal.
ZSSH might be for you
One option that came to me talking this over with a couple of fellas and reading your post. If you want “Johnny Admin” to be able to SSH to the iot gateway (shown in green on the image), well zssh is literally all you would need. zssh is a ‘zitification’ - an app that allows you to ssh to a machine over ziti. It supports this feature you would make use of called “addressable terminators”. Let’s explore zssh for a moment. With zssh you could make ONE service - called “zssh-to-gateway”. That ONE service would then allow people to ‘zssh’ to the ‘identity name’… for exampe you would run this command: zssh remoteUserName@site1. and presto - you are now on that top green box inside that subnet. zssh remoteUserName@site2 and now you’re in the middle box… If that’s all you want then you CAN do this today. The difference is that you’ll be using an ssh implementation we made from go on the ‘left side’. You can watch this short (under 4 minutes) video of me doing that. I’m happy to talk more about that if you like that approach…
You’ll also notice I put orange ‘ziti’ bubbles ON the iot devices. It sounded to me like this isn’t an option. If that is an option do let me know, but I assume it’s not.
In Summary
Right now with a tunneler on the left and on the right (green bubbles) what you want is not currently available. If you were to use a custom ziti app on both sides, well you can do anything you want and if you are only looking to “ssh” from the admin box to the “iot gateway” (in green) well zssh might be all you really want and need all ready for you to use!
Thanks for the excellent response. The diagram you made at the bottom is exactly what we are trying to achieve.
Our IOT devices don’t have the ability to host any Ziti applications unfortunately.
I think that your zssh solution might be perfect as we don’t really want to have to code a custom app and zssh would be an out of the box solution . It would be an ideal solution as we could access each site by name and then access our IOT devices directly via their local IP address which is exactly what we are looking for.
I will have a look at your video and give it a try.
Why can’t your devices be zitified? Not enough horsepower, like Arduinos or similar, or commercial devices you don’t have the access necessary to? Just wondering as a use case thing.
Remember, naming conventions are your friends. Since Ziti can utilize its own internal FQDNs, a little planning upfront can allow a human readable name for services that is specific, such as location.application.enumeration, or whatever makes sense in your environment. It makes it so much easier to implement, audit, troubleshoot, etc.
Our setup has some industrial equipment which are just not suitable for any high level code and others which are not programmable but have TCP/IP interfaces to reach them.
Thank for the comment we will keep it in mind as this would make life a lot easier in managing everything
I have been trying to get this all to work as you suggested but have hit against a problem that I can’t figure out…
When running zssh on my client I get following error:
PS C:\Users\VMware\Desktop\Zssh> .\zssh.exe -s zsshSvc kmr@sshSvcServer -c .\zsshSvcClient.json -d
INFO username set to: kmr
INFO targetIdentity set to: sshSvcServer
INFO connection to edge router using token 7e386151-484a-4b46-8c56-99740af21450
FATAL error when dialing service name zsshSvc. unable to dial service ‘zsshSvc’: dial failed: service Qf8-t8O.8 has no terminators for identity sshSvcServer
I have setup my identities/service with following script:
#establish some variables which are used below
service_name=zsshSvc
client_identity="${service_name}“Client
server_identity=”${service_name}"Server
the_port=22
#create two identities. one host - one client. Only necessary if you want/need them. Skippable if you #already have an identity. provided here to just ‘make it easy’ to test/try
ziti edge create identity device “${server_identity}” -a "${service_name}"ServerEndpoints -o “${server_identity}”.jwt
ziti edge create identity device “${client_identity}” -a "${service_name}"ClientEndpoints -o “${client_identity}”.jwt
#if you want to modify anything, often deleting the configs/services is easier than updating them #it’s easier to delete all the items too - so until you understand exactly how ziti works, #make sure you clean them all up before making a change
ziti edge delete config “${service_name}”-host.v1
ziti edge delete config “${service_name}”-client-config
ziti edge delete service “${service_name}”
ziti edge delete service-policy “${service_name}”-binding
ziti edge delete service-policy “${service_name}”-dialing
ziti edge create config “${service_name}”-host.v1 host.v1 ‘{“protocol”:“tcp”, “address”:“127.0.0.1”,“port”:’"${the_port}"’, “listenOptions”: {“bindUsingEdgeIdentity”:true}}’ #intercept is not needed for zscp/zssh but make it for testing if you like
ziti edge create config “${service_name}”-client-config intercept.v1 ‘{“protocols”:[“tcp”],“addresses”:["’"${service_name}.ziti"’"], “portRanges”:[{“low”:’"${the_port}"’, “high”:’"${the_port}"’}]}’
ziti edge create service “${service_name}” --configs “${service_name}”-client-config,"${service_name}"-host.v1
ziti edge create service-policy “${service_name}”-binding Bind --service-roles ‘@’"${service_name}" --identity-roles ‘#’"${service_name}"‘ServerEndpoints’
ziti edge create service-policy “${service_name}”-dialing Dial --service-roles ‘@’"${service_name}" --identity-roles ‘#’"${service_name}"‘ClientEndpoints’
Thank you very much for providing the steps! Very helpful! The error you are reporting, in my experience, is pretty much always because the identity assigned to the ziti-edge-tunnel on the remote machine is just not running. Looking at your script I can see you are dialing:
@sshSvcServer
I THINK you really want to be dialing
@zsshSvcServer
What’s happening here is you are telling zssh to dial an identity with the NAME of sshSvcServer. But you are making an identity with the name of zsshSvcServer
Thanks Clint that was exactly the problem and have got a step further, but am still stuck unfortunately.
I now get the following error which I haven’t been able to figure out:
PS C:\Users\VMware\Desktop\Zssh> .\zssh.exe -s zsshSvc kmr@zsshSvcServer -c .\zsshSvcClient.json -d
INFO username set to: kmr
INFO targetIdentity set to: zsshSvcServer
INFO connection to edge router using token 7e386151-484a-4b46-8c56-99740af21450
FATAL error dialing SSH Conn: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
I have tried normal SSH’ing from one win10 machine to the other without ziti and that works fine, I am able to login
When using zssh however, using wireshark I get network traffic between my zsshSvcClient and the Ziti Controller/Router machine but nothing seems to passed on through the tunneler to the zsshSvcServer machine.
Not sure if this will be any help in understanding the issue:
Ah… Ok. here is a “decision” we made with zssh that we might need to ‘fix’… You see for zssh we decided to ONLY support certificate based authentication because frankly it was “less work”. That means you cannot enter your password. You’ll need to use an ssh key… (at this time) I will take an action to document that since it’s really not crystal clear
I can whip up a super fast “how to” video for that if it’d be helpful. But if you already know how to use keys for ssh auth - well all you need to do is:
open local terminal
run: cat ~/.ssh/id_rsa.pub (or ~/.ssh/id_ed25519.pub whatever key you want to use)
ssh to the far machine (you’ll have to do this once)
add the key to your authorized_keys:
run: mkdir ~/.ssh
run: cat [copy/paste your PUBLIC key from above here] >> ~/.ssh/authorized_keys
run zssh
You can also choose to specify the key on the zssh command line by running:
what you are hitting on here is the magic of application embedded zero trust and zssh!
With zssh you do not NEED any tunneller. Also it’s important to be clear here because when you use the words “the tunneler” I hear that you are using a “Desktop Edge” tunneler. Such as the Ziti Desktop Edge for Windows (given you’re in powershell). I’m also assuming you legitimately mean this app as “the tunneler” and your’e not referring to zssh as “the tunneler”. Nomenclature here is somewhat hard. I hope we’re on the same page…
With zssh, your traffic will be seen going ONLY to the edge-router and will not be seen on a local TUN device (100.64.0.x if you’re indeed using the ZDEW). So you should definitely see bytes flowing towards your edge router - but that’s it. (and a few bytes to the controller)
IF you want to further talk wireshark and packets on the wire - would you mind making a new forum post on that which is a bit more targetted/easier for future people to search?
Ok, got a few steps forward but have got stuck as the debug information available isn’t so helpful. Is there any way of increasing the debug level for zssh?
THis is my current state, on the client side I am able to login to my ssh server using key based authentication using the standard Windows10 ssh application (both client and server side).
However when using zssh it fails with
PS C:\Users\VMware\Desktop\Zssh> .\zssh.exe -s zsshSvc kmr@zsshSvcServer -c .\zsshSvcClient.json -i C:\Users\VMware.ssh\id_ed25519 --debug
INFO username set to: kmr
INFO targetIdentity set to: zsshSvcServer
INFO connection to edge router using token 7e386151-484a-4b46-8c56-99740af21450
connected.
The system cannot find the path specified.
Whereas on the server side I get
[2022-03-28T11:22:00.409Z] INFO tunnel-cbs:D:\a\ziti-tunnel-sdk-c\ziti-tunnel-sdk-c\lib\ziti-tunnel-cbs\ziti_hosting.c:604 on_hosted_client_connect() hosted_service[zsshSvc], client[zsshSvcClient] incoming connection
boy that’s definitely not “the most helpful error” I’ve ever seen. Sorry about that. What it would appear to be to me is that the ssh key is not being found. You’re certain that is the correct path to the identity file? If you are - my next guess is that we might not be escaping backslashes correctly. I see you provided a full path to that file - maybe try using : c:/Users/VMware.ssh/id_ed25519. Backslashes are notorious for making bugs…
I’ll try adding a few more debug steps in the code and i’ll produce a new build to troubleshoot.
Odd… I just downloaded the 0.0.10 binary and ran it myself. I also used a fully qualified path.
C:\temp\zssh-windows-amd64.exe -s zsshSvc ubuntu@zsshSvcServer -c .\zsshSvcClient.json -i C:\Users\cd\.ssh\my.pem
INFO connection to edge router using token 49bfaa64-0767-497b-ac8b-d0f2f38719cf
WARN could not connect to openssh-ssh-agent pipe, is the ssh-agent service (OpenSSH Authentication Agent) running?
connected.
You an see I get a warning because I don’t run an ssh-agent but it worked…
When i provide a missing key (still using a fully qualified path) I get:
C:\temp\zssh-windows-amd64.exe -s zsshSvc ubuntu@zsshSvcServer -c .\zsshSvcClient.json -i C:\temp\no-key-here
INFO connection to edge router using token 48b23eb5-08ed-4821-929c-16e3499dc7a8
ERROR could not read zssh file [C:\temp\no-key-here]: open C:\temp\no-key-here: The system cannot find the file specified.
WARN could not connect to openssh-ssh-agent pipe, is the ssh-agent service (OpenSSH Authentication Agent) running?
FATAL error dialing SSH Conn: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
Could it be that the “kmr” user doesn’t have a home folder on the remote machine? Probably not I’m guessing…
I’ll keep digging and see if I can come up with anything
Putting the json file and id_ed25519 files locally the outcome is the same.
PS C:\Users\VMware\Desktop\Zssh> .\zssh.exe -s zsshSvc kmr@zsshSvcServer -c zsshSvcClient.json -i id_ed25519 --debug
INFO username set to: kmr
INFO targetIdentity set to: zsshSvcServer
INFO connection to edge router using token 7e386151-484a-4b46-8c56-99740af21450
connected.
The system cannot find the path specified.
If I use windows10 ssh eveything works fine with exactly same setup, so shouldn’t be anything to do with the remote home folder, unless zssh does something differently to windows10 ssh
PS C:\Users\VMware\Desktop\Zssh> ssh kmr@192.168.10.242 -i id_ed25519
Microsoft Windows [Version 10.0.17763.2300]
(c) 2018 Microsoft Corporation. All rights reserved.
kmr@KMR_113124_2 C:\Users\Kmr>
On server side I have checked Eventviewer security logs for
ZSSH login
windows 10 ssh login
and they are almost identical.
Only for the case of zssh, immediately after logging in we have a logoff event
Quite strange, not sure how to go about debugging this as if it works for windows ssh, the same should be true for zssh
. It would be an ideal solution as we could access each site by name and then access our IOT devices directly via their local IP address which is exactly what we are looking for.
