Creating circuit error: how to understand it?

zrok
1

I am not sure about what is happening on routers.

I would say zrok tcptunnel does not work.

{
  "_context": "ch{edge}->u{classic}->i{zID/2xmp}",
  "chSeq": 7,
  "connId": 1,
  "edgeSeq": 0,
  "error": "exceeded maximum [2] retries creating circuit [c/j11o4KaOm] (error creating route for [s/j11o4KaOm] on [r/Y-UUoWCyIW] (error creating route for [c/j11o4KaOm]: timeout waiting for message reply: context deadline exceeded))",
  "file": "github.com/openziti/ziti/router/xgress_edge/listener.go:199",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect",
  "level": "warning",
  "msg": "failed to dial fabric",
  "time": "2025-08-07T19:49:28.113Z",
  "type": "EdgeConnectType"
}

The tcptunnel generates tons of messages

{"time":"2025-08-07T19:49:04.340Z"}
{"time":"2025-08-07T19:49:04.340Z"}
{"time":"2025-08-07T19:49:04.341Z"}
{"time":"2025-08-07T19:49:04.341Z"}
{"time":"2025-08-07T19:49:05.491Z"}
{"time":"2025-08-07T19:49:05.507Z"}
{"time":"2025-08-07T19:49:07.578Z"}
{"time":"2025-08-07T19:49:09.838Z"}
{"time":"2025-08-07T19:49:09.839Z"}
{"time":"2025-08-07T19:49:09.839Z"}
......


{
  "file": "/__w/zrok/zrok/cmd/zrok/shareReserved.go:359",
  "func": "main.(*shareReservedCommand).shareLocal",
  "level": "info",
  "msg": "ziti-edge-router connId=2147485782, logical=ziti-sdk[router=tls:tunnel.svc.host:rport] -> ACCEPT 127.0.0.1:port",
  "time": "2025-08-07T19:49:26.336Z"
}


{
  "file": "/__w/zrok/zrok/endpoints/txer.go:33",
  "func": "github.com/openziti/zrok/endpoints.TXer",
  "level": "error",
  "msg": "read error '127.0.0.1:port' -> 'ziti-edge-router connId=2147486157, logical=ziti-sdk[router=tls:tunnel.svc.host:rport]': read tcp 127.0.0.1:48812->127.0.0.1:port: use of closed network connection",
  "time": "2025-08-07T19:53:17.788Z"
}


journalctl --since "Aug 07 19:49:00" -u zrok-tcptunnel.service  | grep 'use of closed' | wc -l
828

Does zrok tcptunnel have a limit on the number of connections? The user tried to run apt upgrade via local http proxy bound to squid using a zrok tcptunnel. At the same time I tried to use the tunnel, curl – proxy localhost:lport https://www.google.com and got an error. There was a network congestion. The total number of connection was approximately 90.

2

tcp tunnel does not work

{
  "file": "/__w/zrok/zrok/cmd/zrok/shareReserved.go:359",
  "func": "main.(*shareReservedCommand).shareLocal",
  "level": "info",
  "msg": "ziti-edge-router connId=2147489798, logical=ziti-sdk[router=tls:router.name:rport] -> ACCEPT 127.0.0.1:port",
  "time": "2025-08-12T19:24:23.125Z"
}
{
  "circuitId": "9Vo8pLeOn",
  "connId": 1,
  "error": "timeout waiting for message to be written to wire: context deadline exceeded",
  "file": "/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v1.2.1/ziti/edge/network/conn.go:824",
  "func": "github.com/openziti/sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded",
  "level": "error",
  "msg": "failed to send reply to dial request",
  "time": "2025-08-12T19:24:32.129Z"
}
{
  "circuitId": "zMU8pLaRm",
  "connId": 1,
  "error": "timeout waiting for message to be written to wire: context deadline exceeded",
  "file": "/github/home/go/pkg/mod/github.com/openziti/sdk-golang@v1.2.1/ziti/edge/network/conn.go:824",
  "func": "github.com/openziti/sdk-golang/ziti/edge/network.(*newConnHandler).dialSucceeded",
  "level": "error",
  "msg": "failed to send reply to dial request",
  "time": "2025-08-12T19:24:37.850Z"
}

Every time this happens I restart the zrok share to make the tcptunnel work again.

Here is the router’s log

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cme8x26jogj4ucqig56e310fj",
  "attempt": 0,
  "attemptNumber": "1",
  "binding": "edge",
  "circuitId": "9Vo8pLeOn",
  "context": "ch{ctrl}->u{reconnecting}->i{dc/xzYA}",
  "destination": "4K6vU1SVTHHDN4WQGSRKBC",
  "error": "error creating route for [c/9Vo8pLeOn]: timeout waiting for message reply: context deadline exceeded",
  "file": "github.com/openziti/ziti/router/handler_ctrl/route.go:140",
  "func": "github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail",
  "level": "error",
  "msg": "failure while handling route update",
  "serviceId": "4TDIgoL24R7kDi6z3EQZCe",
  "sessionId": "cme8x2784gj4wcqigloudf6qb",
  "time": "2025-08-12T19:24:27.090Z"
}
{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cme8x26jogj4ucqig56e310fj",
  "attemptNumber": "2",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "9Vo8pLeOn",
  "connId": 2147489799,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:132",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "4TDIgoL24R7kDi6z3EQZCe",
  "sessionId": "cme8x2784gj4wcqigloudf6qb",
  "terminatorAddress": "4K6vU1SVTHHDN4WQGSRKBC",
  "time": "2025-08-12T19:24:27.127Z"
}
{
  "file": "github.com/openziti/ziti/router/xgress_edge/accept.go:171",
  "func": "github.com/openziti/ziti/router/xgress_edge.debugPeekHandler.Tx",
  "level": "info",
  "msg": "sending dial: seq: 130170 , connId: 1, newConnId: 2147489799, circuitId: 9Vo8pLeOn",
  "time": "2025-08-12T19:24:27.127Z"
}
3

There is nothing particularly different about tcpTunnel shares versus other types of shares. It does nothing different with regard to circuits on the ziti network.

Can you possibly describe in more detail what you’re trying to accomplish?

4

It simply does not work.

Where I should look?

apt full-upgrade

5

It definitely works. What is your zrok share command? How are you trying to zrok access it?

Are you trying to use tcpTunnel against a public frontend? You cannot access it via http in your public frontend. You must use zrok access private to bind the share to a network listener.

6

If I use zrok share private -b tcpTunnel 192.168.9.1:22 to create a share for the ssh endpoint on one of my local systems… I can then zrok access private <shareToken> that share to bind it to a local network address (127.0.0.1:9191)… and then I can ssh to 127.0.0.1:9191.

That is how tcpTunnel works.

7

apt full-upgrade

It works only then I access via firefox. (small load)

On other hand then users download a lot of things the zrok tcptunnel does not reply to routers.

8

I don’t understand what you’re saying.

9

Then I setup a proxy in FF, zrok tcptunnel works, similar

curl --proxy http://127.0.0.1:port http://www.google.com

Then we run apt full-upgrade zrok tcptunnel stops

10

What zrok command are you using to create your tcpTunnel? What zrok access command are you using to bind the resulting share?

11

It is a reserved share,

zrok share reserved proxy --override-endpont 127.0.0.1:port --headless

As a usual share.

12

And you are binding it to a network address using zrok access how?

13

Is there a way to see it somehow?

I use yours scripts:

env file:

ZROK_TARGET="127.0.0.1:port"

ZROK_BACKEND_MODE="tcpTunnel"

ZROK_PERMISSION_MODE="open"

I should say that It works very well for FF.

The problem appears when someone tries to download a lot of things via apt.

14

zrok access does not care about the target, the backend mode, or the permission mode.

I’m not sure what you’re doing, but I think there is some confusion.

The zrok access private command takes a share token, and optionally a bind address and creates a network listener that routes traffic through the share. I would recommend doing this directly from the CLI and trying that.

Something like zrok access private proxy –bind <address>:<port> and then pointing your traffic at <address>:<port>.

15

Ok. But the problem might be related to your usage pattern. If you’re trying to do something outside of how it’s intended to be used, you might have partial success.

16

As I explained It works pretty well.

The only problem if someone run apt using this tcptunnel.

17

Assuming it’s being used properly… based on your initial post, it sounds like you’re running into some kind of network congestion. There are no limits on the number of simultaneous connections that can be handled through tcpTunnel… but if you’re running into issues on your ziti network, you might be running out of underlay or overlay capacity.

18

And your apt usage scenario might be opening a LOT of connections in parallel. Creating connections can cause overhead on the overlay.

19

Firstly we run the command:

 curl --proxy http://127.0.0.1:port -s -w '%{time_total}s' -o /dev/null http://www.google.com
0.216196s

Then we run

curl --proxy http://127.0.0.1:port -o /dev/null https://sbg.proof.ovh.net/files/10Gb.dat

It will take time….

So we open another terminal window and repeat:

 curl --proxy http://127.0.0.1:port -s -w '%{time_total}s' -o /dev/null http://www.google.com
11.825697s

and then

curl  -s -w '%{time_total}s' -o /dev/null http://www.google.comgoogle.com
0.104771s
20

If you combine significant upload or download traffic, while creating a large number of connections, you could certainly run into timeouts if your underlay is saturated, or your overlay has insufficient capacity.