Ssh via openziti hangs or throw error banner line contains invalid characters

I’ve setup new service ssh with same config as http from quickstart. Http works perfectly, but with ssh there are some problems.
It hangs on this stage

Jun 14 01:32:06 route sshd[117994]: debug1: Forked child 118492.
Jun 14 01:32:06 route sshd[118492]: debug1: Set /proc/self/oom_score_adj to 0
Jun 14 01:32:06 route sshd[118492]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jun 14 01:32:06 route sshd[118492]: debug1: inetd sockets after dupping: 4, 4
Jun 14 01:32:06 route sshd[118492]: Connection from 10.0.0.207 port 48490 on 10.0.0.10 port 22 rdomain “”
Jun 14 01:32:06 route sshd[118492]: debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3

and when i cancel ssh from client it throw this
Jun 14 01:33:52 route sshd[118492]: error: kex_exchange_identification: Connection closed by remote host

on client side I have this logs

OpenSSH_8.2p1 Ubuntu-4ubuntu0.3, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/XXX/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to ssh.ziti [100.64.2.1] port 22.
debug1: Connection established.
debug1: identity file /home/XXX/.ssh/id_rsa type 0
debug1: identity file /home/XXX/.ssh/id_rsa-cert type -1
debug1: identity file /home/XXX/.ssh/id_dsa type -1
debug1: identity file /home/XXX/.ssh/id_dsa-cert type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/XXX/.ssh/id_ed25519 type -1
debug1: identity file /home/XXX/.ssh/id_ed25519-cert type -1
debug1: identity file /home/XXX/.ssh/id_ed25519_sk type -1
debug1: identity file /home/XXX/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/XXX/.ssh/id_xmss type -1
debug1: identity file /home/XXX/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3

after cancellation it may throw error with this on server

Jun 14 01:37:57 route sshd[117994]: debug1: Forked child 118703.
Jun 14 01:37:57 route sshd[118703]: debug1: Set /proc/self/oom_score_adj to 0
Jun 14 01:37:57 route sshd[118703]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jun 14 01:37:57 route sshd[118703]: debug1: inetd sockets after dupping: 4, 4
Jun 14 01:37:57 route sshd[118703]: Connection from 10.0.0.207 port 48494 on 10.0.0.10 port 22 rdomain “”
Jun 14 01:37:57 route sshd[118703]: debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
Jun 14 01:37:57 route sshd[118703]: error: kex_exchange_identification: banner line contains invalid characters
Jun 14 01:37:57 route sshd[118703]: banner exchange: Connection from 10.0.0.207 port 48494: invalid format

with error on client side

OpenSSH_8.2p1 Ubuntu-4ubuntu0.3, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/XXX/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to ssh.ziti [100.64.2.1] port 22.
debug1: Connection established.
debug1: identity file /home/XXX/.ssh/id_rsa type 0
debug1: identity file /home/XXX/.ssh/id_rsa-cert type -1
debug1: identity file /home/XXX/.ssh/id_dsa type -1
debug1: identity file /home/XXX/.ssh/id_dsa-cert type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/XXX/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/XXX/.ssh/id_ed25519 type -1
debug1: identity file /home/XXX/.ssh/id_ed25519-cert type -1
debug1: identity file /home/XXX/.ssh/id_ed25519_sk type -1
debug1: identity file /home/XXX/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/XXX/.ssh/id_xmss type -1
debug1: identity file /home/XXX/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
kex_exchange_identification: Connection closed by remote host

without openziti layer ssh auth work’s fine with no errors.

Hi @Wild , welcome to the OpenZiti community! Thanks for the post.

What tunneling client are you using, ziti-edge-tunnel? All those -1 results seem odd, and "kex_exchange_identification: banner line contains invalid characters” makes me wonder if it’s something to do with end to end encryption.

How did you setup the services? Did you use the cli? Are you comfortable sharing the commands you ran? I’ve run ssh services over OpenZiti, I’m surprised (and sad) it doesn’t also just work for you.

Hello. I use this and works for me. Probably is encryption, Look near the bottom on Issue creating tunnel service doesn’t work in GUI but does through command line - openziti as I had a similar experience publishing another web service. Commands to fix through GUI and command line given here.

@dovholuknf - I got mono configured last night, so might write this up as well (unless ZAC now has ability to flip encryption through GUI)

Hello @dovholuknf, thank you for your reply!
my version of ziti-edge-tunnel is v0.17.7 on Ubuntu 22.04 LTS
I’ve copied them in ziti http-console.
After your reply I tried to recreate all with bash-console commands and everything start working perfect!

@Wild - did you use the GUI initially?

@dovholuknf - we should have equivalency between the GUI and command line. That is, if the command line enables the encryption by default, then the GUI should do the same.

@gooseleggs yes, i’ve used gui for “copy” service and policy settings.
I realised that it’s better do everything with bash console without gui :slight_smile:

@curt has been working on something pretty cool around this. I don’t think it has this exact option but in an upcoming Ziti TV we plan to demonstrate “turning ZAC” dark… Anything you want to add @curt?

100% agree. I filed this issue. I expect we’ll get this fixed quickly Encrypted by default · Issue #33 · openziti/ziti-console · GitHub

the CLI and the UI should behave the same - for sure.