Documentation SSH Access with OpenZiti

[garry0]

After further troubleshooting, I identified that the client identity could not previously ping the host identity (10.252.252.174). Once this issue was resolved, I re-ran the ziti ops verify-traffic command, and the "terminator not found" problem is now gone. Below is the output from the verify-traffic command:

[root@d6c1a056bbe1 ~]# ziti ops verify-traffic --verbose  
WARNING no prefix and mode [] is not 'both'. default prefix of 2024-12-02-0323 will be used  
INFO    generating P-384 EC key  
INFO    generating P-384 EC key  
INFO    waiting 10s for terminator for service: 2024-12-02-0323.verify-traffic  
INFO    successfully bound service: 2024-12-02-0323.verify-traffic  
INFO    Server is listening for a connection and will exit when one is received.  
INFO    new service session                           session token=5d5bbc10-f1ef-4cea-9120-c148f7950b3e  
INFO    found terminator for service: 2024-12-02-0323.verify-traffic  
INFO    found service named: 2024-12-02-0323.verify-traffic  
INFO    Server has accepted a connection and will exit soon.  
INFO    successfully dialed service: 2024-12-02-0323.verify-traffic  
INFO    verify-traffic test successfully detected  
INFO    Server complete. exiting  
INFO    client complete  

Additionally, I captured logs from both the controller and the router during the verify-traffic command execution. Here are the relevant snippets:

ziti-router logs:

Dec 02 10:23:09 almalinuxztna-174 ziti[49538]: {"_context":"{c/Z0HzvJO7t|@/4xqz}\u003cTerminator\u003e","ctrlId":"NetFoundry Inc. Client eV-XglaFC","file":"github.com/openziti/ziti/router/handler_xgress/close.go:66","func":"github.com/openziti/ziti/router/handler_xgress.(*closeHandler).HandleXgressClose","level":"error","msg":"control channel not available","time":"2024-12-02T10:23:09.293Z"}
Dec 02 10:23:10 almalinuxztna-174 ziti[49538]: {"ctrlId":"NetFoundry Inc. Client eV-XglaFC","file":"github.com/openziti/ziti/router/forwarder/faulter.go:101","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run","level":"error","msg":"no control channel for controller","time":"2024-12-02T10:23:10.666Z"}
Dec 02 10:23:30 almalinuxztna-174 ziti[49538]: {"_context":"ch{ctrl}-\u003eu{reconnecting}-\u003ei{k95k}","file":"github.com/openziti/ziti/router/handler_ctrl/validate_terminators_v2.go:94","func":"github.com/openziti/ziti/router/handler_ctrl.(*validateTerminatorsV2Handler).validateTerminators.func1","level":"info","msg":"validating terminator","terminatorId":"6BOlQ1KOsPFwXHaB71LmxE","time":"2024-12-02T10:23:30.390Z"}
Dec 02 10:23:55 almalinuxztna-174 ziti[49538]: {"circuitId":"isvqv3o7t","ctrlId":"NetFoundry Inc. Client eV-XglaFC","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":511484000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-12-02T10:23:55.590Z"}
Dec 02 10:23:55 almalinuxztna-174 ziti[49538]: {"circuitId":"DVxaC3o7wx","ctrlId":"NetFoundry Inc. Client eV-XglaFC","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":983052000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-12-02T10:23:55.590Z"}
Dec 02 10:23:55 almalinuxztna-174 ziti[49538]: {"circuitId":"Ok4vr3oAt","ctrlId":"NetFoundry Inc. Client eV-XglaFC","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":1185676000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-12-02T10:23:55.590Z"}
Dec 02 10:23:55 almalinuxztna-174 ziti[49538]: {"circuitId":"t1rnIJo7w","ctrlId":"NetFoundry Inc. Client eV-XglaFC","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":1757648000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2024-12-02T10:23:55.590Z"}
Dec 02 10:23:55 almalinuxztna-174 ziti[49538]: {"ctrlId":"NetFoundry Inc. Client eV-XglaFC","file":"github.com/openziti/ziti/router/forwarder/scanner.go:105","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","level":"error","msg":"no ctrl channel, cannot request circuit confirmations","time":"2024-12-02T10:23:55.590Z"}  

ziti-controller logs:

Dec 02 10:23:08 almalinuxztna-174 ziti[49587]: {"_context":"ch{jtGKEhO5R}-\u003eu{classic}-\u003ei{k95k}","createTime":2270952,"file":"github.com/openziti/ziti/controller/handler_edge_ctrl/create_terminator_v2.go:173","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*createTerminatorV2Handler).CreateTerminatorV2","level":"info","msg":"created terminator","routerId":"jtGKEhO5R","service":"2024-12-02-0323.verify-traffic","serviceId":"21dGoQFSrvYthKG4O5Hdy5","terminator":"6BOlQ1KOsPFwXHaB71LmxE","terminatorId":"6BOlQ1KOsPFwXHaB71LmxE","time":"2024-12-02T10:23:08.187Z","token":"5d5bbc10-f1ef-4cea-9120-c148f7950b3e"}
Dec 02 10:23:08 almalinuxztna-174 ziti[49587]: {"_context":"ch{jtGKEhO5R}-\u003eu{classic}-\u003ei{k95k}","elapsed":2675744,"file":"github.com/openziti/ziti/controller/handler_edge_ctrl/create_terminator_v2.go:194","func":"github.com/openziti/ziti/controller/handler_edge_ctrl.(*createTerminatorV2Handler).CreateTerminatorV2","level":"info","msg":"completed create terminator v2 operation","routerId":"jtGKEhO5R","service":"2024-12-02-0323.verify-traffic","serviceId":"21dGoQFSrvYthKG4O5Hdy5","terminatorId":"6BOlQ1KOsPFwXHaB71LmxE","time":"2024-12-02T10:23:08.187Z","token":"5d5bbc10-f1ef-4cea-9120-c148f7950b3e"}
Dec 02 10:23:08 almalinuxztna-174 ziti[49587]: {"_context":"ch{jtGKEhO5R}-\u003eu{classic}-\u003ei{k95k}","file":"github.com/openziti/ziti/controller/handler_ctrl/remove_terminators.go:66","func":"github.com/openziti/ziti/controller/handler_ctrl.(*removeTerminatorsHandler).handleRemoveTerminators","level":"info","msg":"removed terminators","routerId":"jtGKEhO5R","terminatorIds":["6BOlQ1KOsPFwXHaB71LmxE"],"time":"2024-12-02T10:23:08.285Z"}
Dec 02 10:23:30 almalinuxztna-174 ziti[49587]: {"file":"github.com/openziti/ziti/controller/network/router_messaging.go:298","func":"github.com/openziti/ziti/controller/network.(*RouterMessaging).sendTerminatorValidationRequest","level":"info","msg":"queuing validate of terminator","terminatorId":"6BOlQ1KOsPFwXHaB71LmxE","time":"2024-12-02T10:23:30.389Z"}
Dec 02 10:23:30 almalinuxztna-174 ziti[49587]: {"file":"github.com/openziti/ziti/controller/network/router_messaging.go:512","func":"github.com/openziti/ziti/controller/network.(*terminatorValidationRespReceived).DeleteInvalid","level":"info","msg":"queuing terminator for delete","reason":"UnknownTerminator","routerId":"jtGKEhO5R","terminatorId":"6BOlQ1KOsPFwXHaB71LmxE","time":"2024-12-02T10:23:30.390Z"}

Given that the "terminator not found" issue is resolved and the verify-traffic command shows a successful result, would you say the problem has been fully addressed, or is there anything else I should check to ensure everything is functioning correctly?

[TheLumberjack]

@garry0

Given that the "terminator not found" issue is resolved and the verify-traffic command shows a successful result, would you say the problem has been fully addressed

Yes, it's highly likely. The only other test is to run the verify traffic command on any other target machine. Sometimes the advertised address works when run on the same machine, but firewalls, routing issues etc can creep in when running elsewhere on a different machine. That's why it's sometimes useful to run the test on any other machine. The command also allows you to start a server on one machine, then run a client on a different machine. (You can check the run mode option to try that if interested)

But, getting the verify traffic command to work, is a really good sign.

@garry0

I identified that the client identity could not previously ping the host identity

Can you elaborate on this? Ping is not something the OpenZiti overlay supports. So I'm not exactly sure what you meant. Since the verify traffic command succeeded, I'm not sure this is relevant, but I'm interested in what you did to fix it, it sounds like something I could maybe add to the verify traffic/network command

[garry0]

I have tested the ziti ops verify-traffic command on another machine, and the output showed no issues. Additionally, I successfully connected via SSH using the DNS t2tssh.ziti. Thank you for helping me resolve this issue.

As for the terminator problem, I fixed it by reinstalling the Ziti router. Below is the command history from my server during troubleshooting:

584  firewall-cmd --list-all
585  firewall-cmd --permanent --add-icmp-block-inversion
586  sudo systemctl reload firewalld
587  sudo systemctl daemon-reload
588  firewall-cmd --list-all
589  firewall-cmd --add-icmp-type=echo-request --permanent
590  firewall-cmd --permanent --add-icmp-type=echo-request
591  firewall-cmd --permanent --add-rich-rule='rule protocol value="icmp" accept'
592  firewall-cmd --reload
593  systemctl daemon-reload
594  firewall-cmd --list-all
595  systemctl status firewalld
596  systemctl stop firewalld
597  systemctl status firewalld
598  systemctl start firewalld
599  systemctl status firewalld
600  sysctl net.ipv4.icmp_echo_ignore_all

[TheLumberjack]

@garry0

Additionally, I successfully connected via SSH using the DNS t2tssh.ziti

Awesome! Glad to hear you got things working, thanks for the follow-up.

[garry0]

I would like to ask about creating services for Prometheus and Grafana with ports 9090 and 3000 in OpenZiti to access their web dashboards. Should the configuration be similar to the SSH service configuration?

[TheLumberjack]

@garry0

Should the configuration be similar to the SSH service configuration?

"Maybe" is the only real answer. If Prometheus and Grafana are running on the exact same computer that you're ssh'ing to, then sure, that'd be just fine.

If you want the same identity to have access, you could change the service to allow all three ports too. Then you'd have one service with access to the three ports. Or you could make one "montioring" service, and grant two ports to it (9090/3000)... Really it's up to you and what you're trying to do.

[garry0]

I successfully created a monitoring service to access the Prometheus dashboard on port 9090, and it works perfectly. However, when I tried adding port 3000 to the same service to access the Grafana dashboard, I encountered the following error in the Ziti Tunneler log:

[2024-12-10T05:05:15.106Z]   ERROR ziti-sdk:connect.c:1069 connect_reply_cb() conn[0.0/gr1tUcOz/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/i4eW.VOAw]: error creating route for [s/i4eW.VOAw] on [r/jtGKEhO5R] (error creating route for [c/i4eW.VOAw]: timeout waiting for message reply: context deadline exceeded)
[2024-12-10T05:05:15.106Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed
[2024-12-10T05:05:25.626Z]   ERROR ziti-sdk:connect.c:1069 connect_reply_cb() conn[0.1/yStjiWJb/Connecting] failed to connect, reason=exceeded maximum [2] retries creating circuit [c/njkWDVOAw]: error creating route for [s/njkWDVOAw] on [r/jtGKEhO5R] (error creating route for [c/njkWDVOAw]: timeout waiting for message reply: context deadline exceeded)
[2024-12-10T05:05:25.626Z]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:103 on_ziti_connect() ziti dial failed: connection is closed

Did I miss anything in the service configuration or policies?

[TheLumberjack]

Network services can be tricky. There's a ton of different reasons this could happen. My best guesses in order of likelihood are:

• Port 9090 isn't actually running
• Grafana is bound to an IP other than 127.0.0.1 or 0.0.0.0 (assuming you're using 127.0.0.1 as the offload address)
• You used localhost as the offload and somehow grafana is listening on an IPv4 address, but localhost used IPv6 instead

You should ssh to the machine and run the same curl as you configured the service to offload from ziti and verify curl works.

I expect it's one of the first two, the last one is quite unlikely.

Also check firewalls etc. hth

[Tetrov]

@TheLumberjack

If you find you need to ssh to many, many machines there are other techniques you might want to use. If you have only a few machines to ssh into, either guide seems fine to me.

@TheLumberjack what are the "other techniques" please?

[bmullan]

Parallel SSH is easy to use

PSSH - Run Commands on Multiple Remote Linux Systems

[TheLumberjack]

Hi @Tetrov, welcome to the community and to OpenZiti (and zrok/BrowZer)!

The other techniques I alluded to are based around leveraging an OpenZiti concept called addressable terminators. Using addressable terminators will allow you to effectively use the name of an identity as the target of the service. This allows one to declare a single service and have it be usable/bindable by multiple identities, instead of having 1 service per identity.

It's been discussed in the past in the forum, I'll try to find it and reference it here

[TheLumberjack]

Here's one post that came to mind

OpenZiti host level overlay

Most of the Ziti tunnelers also support a couple of “tricks” that make this kind of thing a little easier. The first one that comes to mind after seeing Clint’s examples is using forwardPort in the host.v1 configuration. Port Forwarding ziti edge create config "ubussh.host.v1" host.v1 '{"protocol":"tcp", "address":"localhost","forwardPort": true, "allowedPortRanges":[{"low":1, "high":16384}] }' You would use this in combination with a port range in the intercept.v1 configuration to make the …

And this one

Binding a service to multiple hosts

First off - I have given this a go as it appears to make sense in my head, but not when I give it a go. The outcome, is that I will likely have a number of 1-1 mappings of users to their desktop machines at work from their home device. In my mind, I think I should be able to create one service, and through service policies apply this to different identity bind/dial configurations, but this does not seem to work. I always end up on the first configured (OK, I have only defined two - one workin…

I remember these posts at excellent from @scareything . Hopefully these help with the understanding but if not, let us know.