Summary
SSH connections through OpenZiti hang immediately after TCP handshake. The client never receives the SSH banner from the server, despite sshd successfully sending it to the local ZET socket.
Environment
-
Server (host): Ubuntu, ziti-edge-tunnel v1.7.11, Ziti SDK v1.8.3
-
Client: macOS, Ziti Desktop Edge (ZDE)
-
Controller: v1.6.7
-
Edge Router: v1.6.7
-
Service Config:
host.v1withaddress: 127.0.0.1, port: 22
Symptoms
-
DNS resolution works:
vault.zitiresolves correctly -
TCP connect succeeds:
nc -vz vault.ziti 22returns success -
SSH hangs immediately after connection:
debug1: Connecting to vault.ziti port 22.
debug1: Connection established.
debug1: Local version string SSH-2.0-OpenSSH_10.0
[HANGS FOREVER - NO SERVER BANNER]
What We Found
ZET Logs Show Connection Accepted
INFO tunnel-cbs:ziti_hosting.c:706 on_hosted_client_connect() hosted_service[vault-ssh] client[ssh-client] client_src_addr[tcp:100.100.0.1:52636] dst_addr[tcp:127.0.0.1:22]: incoming connection
ZET accepts the connection and forwards to local sshd.
tcpdump on Loopback Shows Banner Sent
06:30:03.534162 IP 127.0.0.1.22 > 127.0.0.1.60158: Flags [P.], seq 1:44, ack 1, win 512, length 43:
SSH: SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.13
sshd successfully sends the SSH banner to ZET's local socket.
Active Connections Show Data Not Forwarded
CLOSE-WAIT 0 0 127.0.0.1:60158 127.0.0.1:22 users:(("ziti-edge-tunne",pid=32975,fd=23))
ZET has an active connection to sshd with the banner received.
Mac Client Never Receives Banner
The SSH client on Mac never receives the SSH-2.0-OpenSSH... banner from the server, causing the connection to hang indefinitely.
Root Cause Analysis
ZET successfully:
-
Accepts incoming Ziti tunnel connection from client
-
Opens local TCP connection to 127.0.0.1:22
-
Receives SSH banner from sshd on loopback
ZET FAILS to:
- Forward the server→client data (SSH banner) back through the Ziti tunnel to the remote client
This appears to be a bidirectional data forwarding issue specifically affecting server-first protocols (SSH, FTP, SMTP) where the server must send initial data before the client speaks.
Steps to Reproduce
-
Configure ZET v1.7.11 on Ubuntu server with
host.v1service for SSH (port 22) -
Configure ZDE on macOS client
-
Attempt SSH connection:
ssh -vvv vault.ziti -
Observe: Connection hangs after "Local version string" - no banner received
-
On server, run:
sudo tcpdump -i lo -A 'tcp port 22 and host 127.0.0.1' -
Observe: sshd sends banner to ZET's local socket, but client never receives it
Expected Behavior
The SSH banner sent by sshd should be forwarded through the Ziti tunnel to the client, allowing the SSH handshake to proceed normally.
Actual Behavior
The banner is received by ZET but never forwarded to the client, causing SSH to hang indefinitely.
Configuration
Service (host.v1)
json
{
"protocol": "tcp",
"address": "127.0.0.1",
"port": 22
}
ZET Service Status
INFO tunnel-cbs:ziti_tunnel_ctrl.c:976 on_service() hosting server_address[tcp:127.0.0.1:22] service[vault-ssh]
Service is properly bound and accepting connections.
Additional Notes
-
Other services work: vault.engchat.app on port 8200 (HTTP) works correctly through the same ZET instance
-
Local SSH works: Direct SSH to 127.0.0.1:22 on server works perfectly
-
TCP connectivity works:
nc -vz vault.ziti 22succeeds, proving tunnel connectivity -
No firewall issues: All traffic is on loopback (127.0.0.1) for the server side
-
No errors in ZET logs: Only shows
on_hosted_client_connect(), no subsequent data transfer or error messages
Questions
-
Is this a known issue in ZET v1.7.11 / SDK v1.8.3?
-
Has this been fixed in newer versions (v2.0.0-alpha)?
-
Is there a workaround configuration (e.g.,
forwardProtocol, different binding mode)? -
Should I enable DEBUG logging to capture more detail about the data path?
Diagnostic Files Available
I have comprehensive diagnostic captures including:
-
Full ZET logs showing connection acceptance
-
tcpdump proving sshd sends banner to ZET
-
Active connection states showing CLOSE-WAIT
-
Network topology and routing tables
Happy to provide any additional information needed to diagnose this issue.
This completely blocks SSH access through OpenZiti for server-first protocols, requiring fallback to direct IP connections which defeats the purpose of zero-trust networking.
any help can be provided or any explanation?
Until I see those logs, I don’t have much to go on.