what you are hitting on here is the magic of application embedded zero trust and zssh!
With zssh you do not NEED any tunneller. Also it's important to be clear here because when you use the words "the tunneler" I hear that you are using a "Desktop Edge" tunneler. Such as the Ziti Desktop Edge for Windows (given you're in powershell). I'm also assuming you legitimately mean this app as "the tunneler" and your'e not referring to zssh as "the tunneler". Nomenclature here is somewhat hard. I hope we're on the same page...
With zssh, your traffic will be seen going ONLY to the edge-router and will not be seen on a local TUN device (100.64.0.x if you're indeed using the ZDEW). So you should definitely see bytes flowing towards your edge router - but that's it. (and a few bytes to the controller)
IF you want to further talk wireshark and packets on the wire - would you mind making a new forum post on that which is a bit more targetted/easier for future people to search?
Ok, got a few steps forward but have got stuck as the debug information available isn’t so helpful. Is there any way of increasing the debug level for zssh?
THis is my current state, on the client side I am able to login to my ssh server using key based authentication using the standard Windows10 ssh application (both client and server side).
However when using zssh it fails with
PS C:\Users\VMware\Desktop\Zssh> .\zssh.exe -s zsshSvc kmr@zsshSvcServer -c .\zsshSvcClient.json -i C:\Users\VMware.ssh\id_ed25519 --debug
INFO username set to: kmr
INFO targetIdentity set to: zsshSvcServer
INFO connection to edge router using token 7e386151-484a-4b46-8c56-99740af21450
connected.
The system cannot find the path specified.
Whereas on the server side I get
[2022-03-28T11:22:00.409Z] INFO tunnel-cbs:D:\a\ziti-tunnel-sdk-c\ziti-tunnel-sdk-c\lib\ziti-tunnel-cbs\ziti_hosting.c:604 on_hosted_client_connect() hosted_service[zsshSvc], client[zsshSvcClient] incoming connection
boy that's definitely not "the most helpful error" I've ever seen. Sorry about that. What it would appear to be to me is that the ssh key is not being found. You're certain that is the correct path to the identity file? If you are - my next guess is that we might not be escaping backslashes correctly. I see you provided a full path to that file - maybe try using : c:/Users/VMware.ssh/id_ed25519. Backslashes are notorious for making bugs...
I'll try adding a few more debug steps in the code and i'll produce a new build to troubleshoot.
Odd… I just downloaded the 0.0.10 binary and ran it myself. I also used a fully qualified path.
C:\temp\zssh-windows-amd64.exe -s zsshSvc ubuntu@zsshSvcServer -c .\zsshSvcClient.json -i C:\Users\cd\.ssh\my.pem
INFO connection to edge router using token 49bfaa64-0767-497b-ac8b-d0f2f38719cf
WARN could not connect to openssh-ssh-agent pipe, is the ssh-agent service (OpenSSH Authentication Agent) running?
connected.
You an see I get a warning because I don’t run an ssh-agent but it worked…
When i provide a missing key (still using a fully qualified path) I get:
C:\temp\zssh-windows-amd64.exe -s zsshSvc ubuntu@zsshSvcServer -c .\zsshSvcClient.json -i C:\temp\no-key-here
INFO connection to edge router using token 48b23eb5-08ed-4821-929c-16e3499dc7a8
ERROR could not read zssh file [C:\temp\no-key-here]: open C:\temp\no-key-here: The system cannot find the file specified.
WARN could not connect to openssh-ssh-agent pipe, is the ssh-agent service (OpenSSH Authentication Agent) running?
FATAL error dialing SSH Conn: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
Could it be that the “kmr” user doesn’t have a home folder on the remote machine? Probably not I’m guessing…
I’ll keep digging and see if I can come up with anything
Putting the json file and id_ed25519 files locally the outcome is the same.
PS C:\Users\VMware\Desktop\Zssh> .\zssh.exe -s zsshSvc kmr@zsshSvcServer -c zsshSvcClient.json -i id_ed25519 --debug
INFO username set to: kmr
INFO targetIdentity set to: zsshSvcServer
INFO connection to edge router using token 7e386151-484a-4b46-8c56-99740af21450
connected.
The system cannot find the path specified.
If I use windows10 ssh eveything works fine with exactly same setup, so shouldn’t be anything to do with the remote home folder, unless zssh does something differently to windows10 ssh
PS C:\Users\VMware\Desktop\Zssh> ssh kmr@192.168.10.242 -i id_ed25519
Microsoft Windows [Version 10.0.17763.2300]
(c) 2018 Microsoft Corporation. All rights reserved.
kmr@KMR_113124_2 C:\Users\Kmr>
On server side I have checked Eventviewer security logs for
ZSSH login
windows 10 ssh login
and they are almost identical.
Only for the case of zssh, immediately after logging in we have a logoff event
Quite strange, not sure how to go about debugging this as if it works for windows ssh, the same should be true for zssh
with ZSSH, I find the following in the sshd logs
8140 2022-03-29 13:50:15.684 debug1: Executing command: “c:\windows\system32\cmd.exe” /c “/bin/bash” with pty
Whereas for Windows10 ssh the following is the corresponding line
2964 2022-03-29 13:57:48.844 debug1: Executing command: “c:\windows\system32\cmd.exe” with pty
If you have control over the sshd server you could increase the logging from it. The openssh server shows you what’s going on in the session if you run it with -ddd. You can also run it from the command line if you pass -D (you’ll want to stop your sshd service if you do this to avoid the port conflict). The following example assumes you’re using the openssh server that comes with Windows.
C:\WINDOWS\system32>c:\Windows\System32\OpenSSH\sshd.exe -D -ddd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 253
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 253
debug3: __PROGRAMDATA__\\ssh/sshd_config:38 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: __PROGRAMDATA__\\ssh/sshd_config:79 setting Subsystem sftp sftp-server.exe
debug3: checking syntax for 'Match Group administrators'
debug1: sshd version OpenSSH_for_Windows_8.1, LibreSSL 3.0.2
debug1: get_passwd: LookupAccountName() failed: 1332.
debug1: private host key #0: ssh-rsa SHA256:cwGZolj5Kajdv5sRACAEKkUIWl/wEw7JxKMdWfXWU+E
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:niWwGcJQqhwp+//XkGmq7oLnsed31VRbX9JaT+ay554
debug1: private host key #2: ssh-ed25519 SHA256:hmIUnpHLPsKckE/5TjO7tEO+fJ9uw28MBRvgAPDs6KE
debug1: rexec_argv[0]='c:\\Windows\\System32\\OpenSSH\\sshd.exe'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
When a client connects you’ll see something like this:
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 253
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
Connection from 10.211.55.2 port 63134 on 10.211.55.5 port 22
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.6
debug1: match: OpenSSH_8.6 pat OpenSSH* compat 0x04000000
…followed by the host and user authentication and any commands that are being spawned for the session. I’m not sure what you’ll see in your logs, but hopefully they offer some clues about the file that is not found.
So this remote machine probably doesn’t have bash on it and ssh is correctly handling that. I’ll have to talk to someone wit more linux knowhow than I have and figure out what the ‘right’ way to deal with this is.
Thanks @Danieleb ! I’ll see if we can address this!
The modification works perfectly and now can reach the IOT gateway (Win10), thanks for getting it up and working so fast.
The next issue on my list is now how to reach the IOT devices since as far as I can see ZSsh doesn’t support port forwarding and I really need to access multiple IPs behind the gateway which need to be accessed from proprietary software… I guess I will need to go back to the drawing board
ProxyJump (-J) support and “remote command” support is on my very near “todo” list for zssh. I should have clarified exactly what you were trying to do. I thought you going to run ssh commands from this “bastion” or subsequently ssh from there explicitly. Not ideal but i thought it would work fine. I take it this machine you zsshed into doesn’t have the proper certificate to let you ssh from it to the target machines?
If zssh had proxyjump support - do you think that’d solve your problem entirely? If yes - I think we can get cracking on implementing it. It’s on my list - but you having a need for it (assuming it’ll address your issue?) would help me prioritize it higher…
One other question - are these iot devices always deployed the exact same way? Is “iot device 1” always192.168.1.2, is “iot device 2” always192.168.1.3, etc?
If that’s the case, there might be other options to consider… We’ll talk about it more internally and see if there’s another option for you in this scenario.
Following with interest. I’d be interested in working on zssh features at some point. Do I understand what you need?
By far the most flexible way to address multiple remote ports through an SSH server is DynamicForward (-D LOCAL_PORT) in lieu of LocalForward (-L LOCAL_PORT:REMOTE_DOMAIN:REMOTE_PORT). However, this sidesteps the way I understand zssh works today with remote identities vs remote domain name.
DynamicForward works similarly to ProxyJump ( e.g. shortcut -J JUMP_USER@JUMP_BOX:PORT) in that it’s proxy-based and therefore unnecessary to bind any additional local ports no matter how many destinations you have. However, this does require that the client side is proxy-aware or has its TCP wrapped with a SOCKS5 provider like proxychains or tsocks. I’m wondering if we should support something like this in zssh.
On one hand it’s an extra step to use things through a proxy, but then so is setting up an identity on every remote so that the intermediate / jump boxen are unnecessary in the first place. Which way are you leaning on this?
