Hi community,
we are experiencing an issue regarding our remote backups which uses a ziti overlay network to connect the source to our remote backup target. Currently, every scheduled backup we run fails.
We are using the promox-backup-client cli tool to backup zfs-snapshots to a virtualized Promox Backup Server in a separate location. The locations are connected via public internet (Germany) using private household internet connections.
This setup used to work, when the amount of data to be backuped was small (<20 GiB). After the dataset size increased to approx. 560 GiB we are continuously experiencing the following issue (promox-backup-client logs)
HTTP/2.0 connection failed
unfinished encoder state dropped
finished encoder state with errors
unclosed encoder dropped
closed encoder dropped with state
unfinished encoder state dropped
catalog upload error - channel closed
Error: stream closed because of a broken pipe
In the relevant logs on the source machine (pve_hydrogen.log) there is a ziti-edge-tunnel installed. We observe lines like these:
Jul 12 03:35:33 hydrogen ziti-edge-tunnel[2020]: (2020)[ 244105.375] ERROR ziti-sdk:channel.c:688 latency_timeout() ch[0] no read/write traffic on channel since before latency probe was sent, closing channel
Jul 12 03:35:33 hydrogen ziti-edge-tunnel[2020]: (2020)[ 244105.375] WARN ziti-sdk:channel.c:852 on_channel_close() ch[0] disconnected from edge router[private0] -110(connection timed out)
Jul 12 03:35:33 hydrogen ziti-edge-tunnel[2020]: (2020)[ 244105.375] ERROR ziti-sdk:connect.c:1049 connect_reply_cb() conn[1.118394/A2lfGNaY/Connecting](pbs-hydrogen.svc) failed to connect [-20/operation did not complete in time]
Jul 12 03:35:44 hydrogen ziti-edge-tunnel[2020]: (2020)[ 244115.568] ERROR tunnel-sdk:tunnel_tcp.c:191 on_tcp_client_err() client=tcp:100.64.0.1:48310 err=-14, terminating connection
On the target machine (pbs_helium.log) there is a ziti-edge-tunnel installed. We observe:
Jul 12 05:35:36 pbs-helium ziti-edge-tunnel[195]: (195)[ 244211.073] WARN ziti-sdk:channel.c:403 on_channel_send() ch[0] write delay = 5.293 q=29 qs=526400
Jul 12 05:35:31 pbs-helium ziti-edge-tunnel[195]: (195)[ 244205.671] ERROR ziti-sdk:connect.c:1477 process_edge_message() conn[1.49358/Pf23Und1/Accepting](pbs-helium.svc) failed to connect, reason=close called
On the “first” router (private0_hydrogen.log) to which the source machine connects:
Jul 12 05:35:39 private0 ziti[123]: {"_context":"ch{edge}->u{classic}->i{ziti-sdk-c[0]@hydrogen/Zylw}","error":"write tcp 10.x.y.z:3022->192.a.b.c:42566: write: connection reset by peer",..."msg":"write error",...}
Jul 12 05:35:20 private0 ziti[123]: {... "error":"error creating route for [c/43sDS3qFenLAxpoIa90kTa]: timeout waiting for message reply: context deadline exceeded",..."msg":"failure while handling route update",...}
Jul 12 05:34:36 private0 ziti[123]: {... "level":"warning","msg":"closing while buffer contains unacked payloads","payloadCount":2,...}
On the “last” router (private1_helium.log) which connects with the target machine:
Jul 12 03:34:31 private1 ziti[129]: {"circuitCount":5,"ctrlId":"atlas-ctrl-client","file":"github.com/openziti/ziti/router/forwarder/faulter.go:107","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run","level":"warning","msg":"reported forwarding faults","time":"2026-07-12T03:34:31.323Z"}
Jul 12 03:34:31 private1 ziti[129]: {"circuitId":"4KRh8X9sjo0t1ryh94FK19","file":"github.com/openziti/ziti/router/forwarder/forwarder.go:155","func":"github.com/openziti/ziti/router/forwarder.(*Forwarder).Unroute","level":"info","msg":"circuit unrouted","time":"2026-07-12T03:34:31.346Z"}
Jul 12 03:34:35 private1 ziti[129]: {"_context":"{c/HnyMaDyXq4UU8AciIX0C8|@/6q65xsfekE08jWiJ2ll67J}\u003cInitiator\u003e","circuitId":"HnyMaDyXq4UU8AciIX0C8","error":"cannot forward payload, no destination for circuit=HnyMaDyXq4UU8AciIX0C8 src=6q65xsfekE08jWiJ2ll67J dst=5WvGiniQ0JG3TsRVoSf7D9","file":"github.com/openziti/ziti/router/handler_xgress/data_plane.go:58","func":"github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload","level":"error","msg":"unable to forward payload","origin":0,"seq":9,"time":"2026-07-12T03:34:35.284Z"}
On “in-between” routers (public0_hydrogen.log & public1_helium.log) on the overlay network we see:
Jul 12 03:34:36 public0 ziti[685]: {"circuitId":"2VSvrd7ZMFcATcHhzcICdD","ctrlId":"atlas-ctrl-client","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":69532000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2026-07-12T03:34:36.773Z"}
We are sharing the logs of all “stations” in the ziti overlay network. We anonymized them to the best of our knowledge, if you see any sensitive data we forgot to redact, please send us a private message so we can take the info down.
As this issue is occurring every night, we were able to visualize some data. The “HTTP/2.0 connection failed” error usually occurs right before 20 GiB transferred are reached just before the one hour mark.
One of things we have tried (not part of the visualization), is to rate limit the transfer/upload speed of the promox-backup-client to --rate 2875KB and --burst 128KB. With these settings we still observe the "HTTP/2.0 connection failed” error at about 175 GiB transferred at the 16h mark. Note that the --rate parameter is set to approx. 40% uplink speed of the source servers internet connection. (The uplink is the bottleneck in our scenario.)
Question:
We cannot get the backups to complete without running into the error and we suspect that it has to do with the ziti overlay (or at least the interplay between promox-backup-client and ziti).
As we are at a loss as how to further troubleshoot this issue, we are happy for any ideas and suggestions. Let us know if you require any further information like network architecture, etc. Maybe someone had a similar issue or just knows a solution to our problem.
Than you very much!
public1_helium.log (2.1 KB)
pbs_helium.log (16.8 KB)
pve_hydrogen.log (28.2 KB)
public0_hydrogen.log (2.6 KB)
private1_helium.log (122.7 KB)
private0_hydrogen.log (66.3 KB)
