Unable to forward payload

Ziti Overlay
1

With ziti-router 1.6.7 I have too much errors. Thus the network is slow.

{
  "_context": "{c/4CpuIMbl75baXLw6iK4v3J|@/7ADl07XYWjSjImnvZaTi05}<Initiator>",
  "circuitId": "4CpuIMbl75baXLw6iK4v3J",
  "error": "cannot forward payload, no forward table for circuit=4CpuIMbl75baXLw6iK4v3J src=7ADl07XYWjSjImnvZaTi05",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 0,
  "seq": 12,
  "time": "2025-09-10T17:25:36.019Z"
}

{
  "_context": "{c/1rpw2FNucpisqlQkWXcql9|@/6Cey3O5tmGwnvK6eoV0llZ}<Initiator>",
  "circuitId": "1rpw2FNucpisqlQkWXcql9",
  "error": "cannot forward payload, no forward table for circuit=1rpw2FNucpisqlQkWXcql9 src=6Cey3O5tmGwnvK6eoV0llZ",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 0,
  "seq": 12,
  "time": "2025-09-10T17:26:36.048Z"
}

{
  "_context": "{c/3KUKd4O50bmdHB6omCfHmg|@/74Z9Ibtv32clR4D1DN1I2t}<Initiator>",
  "circuitId": "3KUKd4O50bmdHB6omCfHmg",
  "error": "cannot forward payload, no forward table for circuit=3KUKd4O50bmdHB6omCfHmg src=74Z9Ibtv32clR4D1DN1I2t",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 0,
  "seq": 12,
  "time": "2025-09-10T17:27:36.115Z"
}
2

Are the errors themselves the problem, or is the logging of errors slowing things down? If you set the log level to fatal, does that resolve the issue?

You use the following to set the log level at runtime.

ziti agent set-log-level fatal -t router

Paul

3

The errors are problematic because this slows down the services.

As the result the network 1.6.7 is slow compared to 1.5.4

Routers 1.5.4 work fine with ziti-controller 1.6.7. => Thus the problem is in the router’s code/configuration.

4

The errors only happen after a circuit is complete and is being torn down. So they are not slowing down services, unless the logging is being overwhelmed by an excess of messages.

5

This is the design :slight_smile:

Unfortunately the ziti-router 1.6.7 generates much more errors, approximately by a factor of several hundreds. This is why the services are slow compared to 1.5.4

They both work with ziti-controller 1.6.7.

The only way to get back to normal is to downgrade the ziti-router to 1.5.4

6
7

There is some misunderstanding: hiding errors will not be helpful.

I am trying to understated why router 1.5.4 runs smoothly but 1.6.7 has such difficulty to handle the payload without disruption.

Clearly every host can handle these additional syslog messages without any noticeable impact on performance. But these errors lead to retransmission/permanent data loss. This is why the services are slower.

8

I can try and reproduce this, but i need to know:

  1. What ziti component is hosting the the service (ER/T, ZET, SDK (which sdk))?
  2. What ziti component is on the client side (ER/T, ZET, SDK (which sdk))?
  3. What does the traffic look like? If you want to be specific about what software is going over Ziti, that's helpful, but need to know protocol, traffic patterns, etc. Is it TCP/UDP? Is it HTTP/SSH, etc? Are you doing request/response/close or is it back and forth? Are you using TCP half-close?
  4. Can you quantify the issue? What throughput/latency are using seeing on 1.5.4 vs 1.6.7? Can you grab metrics and compare retransmission rates between the two? Are you seeing connections be unexpected terminated?

If you can provide specific instructions on how to reproduce the errors, that would be the most helpful, but if you can describe the data flows in detail, I may be able to reproduce the issue.

I did weeks of data flow testing before we released 1.6.7, and the test cases I have are working fine, so we need to figure what's different about your network traffic.

9

Surely, the team does a great job. Thank you!

Problem with performance might have a different root:

As you have explained above the error "unable to forward payload” arises when the service close connection.

Clearly a suffocating client open a bunch of connections in hope to get any reply from the service. But zrok shares are mainly single user services.

Firstly, multiplying connections does not help to get any response from the service.

Secondary, these connections create excess of circuits.

Next, the service will close these connections very soon after.

Finally, the router sends "unable to forward payload” into syslog.

As a solution I try to clone the zrok reserved shares so each user works with its own service.

10

Ok, we're making some progress on understanding the scenario.

So we've got zrok on the front-end and back-end.

  1. Are you running lots of connections over the same front-end and back-end or are you just loading down the one or the other?
  2. You are self-hosting zrok, correct?
  3. Can you either tell me what software you're running over zrok, or describe the traffic patterns? How much traffic is getting send in each direction, is it request/response or is it uncoupled, how large are payloads, etc,etc.
  4. Do you have the 'superNetwork' setting set to true?
11

Yes, you are right I run self-hosted zrok. There is a variety of applications: zrok http proxy, tcp tunneling, socks, vpn.

After your explanation I have done more tests. Sometime there is a relatively large number of connections to the same service. So zrok service can not handle the load.

I don’t use superNetwork.

As I understand this why the router sends a large number of errors:“unable to forward payload”.

The router simply saying that the service gives up and close the connection.

Users see that the service is slow.

12 
curl --proxy http://127.0.0.1:8181 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso

Interrupt the download by sending Ctrl-C

Look at the ziti-router log

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfqn1eboz83ckkig8goy7oqm",
  "attemptNumber": "1",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "6EpRosv59zxcthJlyTZT6Z",
  "connId": 2147483650,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:142",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "4TDIgoL24R7kDi6z3EQZCe",
  "sessionId": "cmfqn1ehbz83ekkigxk8qke50",
  "terminatorAddress": "2sP7UgLnGQkZ4P1nzK6Tjf",
  "time": "2025-09-19T10:24:40.079Z"
}
{
  "_context": "{c/6EpRosv59zxcthJlyTZT6Z|@/7Bttxc3WDL1SZw3znO9NSj}<Terminator>",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/link_send_buffer.go:296",
  "func": "github.com/openziti/sdk-golang/xgress.(*LinkSendBuffer).run",
  "level": "warning",
  "msg": "closing while buffer contains unacked payloads",
  "payloadCount": 106,
  "time": "2025-09-19T10:25:17.037Z"
}
{
  "_context": "{c/6EpRosv59zxcthJlyTZT6Z|@/7Bttxc3WDL1SZw3znO9NSj}<Terminator>",
  "circuitId": "6EpRosv59zxcthJlyTZT6Z",
  "error": "cannot forward payload, no forward table for circuit=6EpRosv59zxcthJlyTZT6Z src=7Bttxc3WDL1SZw3znO9NSj",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 4991,
  "time": "2025-09-19T10:25:17.037Z"
}
{
  "_context": "{c/6EpRosv59zxcthJlyTZT6Z|@/7Bttxc3WDL1SZw3znO9NSj}<Terminator>",
  "error": "write closed",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/xgress.go:870",
  "func": "github.com/openziti/sdk-golang/xgress.(*Xgress).forwardPayload",
  "level": "error",
  "msg": "failure to buffer payload",
  "time": "2025-09-19T10:25:17.037Z"
}
{
  "circuitCount": 1,
  "ctrlId": "dc",
  "file": "github.com/openziti/ziti/router/forwarder/faulter.go:107",
  "func": "github.com/openziti/ziti/router/forwarder.(*Faulter).run",
  "level": "warning",
  "msg": "reported forwarding faults",
  "time": "2025-09-19T10:25:27.244Z"
}
{
  "circuitId": "6EpRosv59zxcthJlyTZT6Z",
  "file": "github.com/openziti/ziti/router/forwarder/forwarder.go:155",
  "func": "github.com/openziti/ziti/router/forwarder.(*Forwarder).Unroute",
  "level": "info",
  "msg": "circuit unrouted",
  "time": "2025-09-19T10:25:27.284Z"
}

After this scenario the service is dead!

curl --proxy http://127.0.0.1:8181 https://www.google.com:443

Ziti-router log

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfqn1eboz83ckkig8goy7oqm",
  "attemptNumber": "1",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "3aeM1fdflUWrVGRLMoGXt4",
  "connId": 2147483651,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:142",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "4TDIgoL24R7kDi6z3EQZCe",
  "sessionId": "cmfqn1ehbz83ekkigxk8qke50",
  "terminatorAddress": "2sP7UgLnGQkZ4P1nzK6Tjf",
  "time": "2025-09-19T10:25:40.521Z"
}
{
  "_context": "{c/3aeM1fdflUWrVGRLMoGXt4|@/eCHiMoFy0coITH9TiNju1}<Terminator>",
  "circuitId": "3aeM1fdflUWrVGRLMoGXt4",
  "error": "cannot forward payload, no forward table for circuit=3aeM1fdflUWrVGRLMoGXt4 src=eCHiMoFy0coITH9TiNju1",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 0,
  "time": "2025-09-19T10:25:45.522Z"
}

Moreover, after I have stopped the zrok share I continue to see the terminator!

 ziti edge list terminators 'limit none' | grep proxy
│ 2sP7UgLnGQkZ4P1nzK6Tjf │ proxy        │ rt89  │ edge    │ 2sP7UgLnGQkZ4P1nzK6Tjf │          │    0 │ default    │            0 │

ziti edge list services 'limit none' | grep proxy
│ 4TDIgoL24R7kDi6z3EQZCe │ proxy                   │ true       │ smartrouting        │                         │

zrok v1.1.2

ziti 1.6.7

You can do the same test using socks5.

curl --proxy socks5://127.0.0.1:9191 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso

Press Ctrl-C

curl -I --proxy socks5://127.0.0.1:9191 https://www.google.com:443
curl: (97) Recv failure: Connection reset by peer

Look at ziti-router log

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfqysannznn9kkig0edtiqzr",
  "attemptNumber": "1",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "2dapZPkNugqIetRNcC9VcP",
  "connId": 2147483705,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:142",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "sessionId": "cmfqysatgznnbkkig5n9v78q8",
  "terminatorAddress": "5B2U1sBaDiWVOjLGoPaOE7",
  "time": "2025-09-19T15:05:34.240Z"
}


{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfqysannznn9kkig0edtiqzr",
  "attemptNumber": "1",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "5OiIr2uYNf35iCLMvlz21a",
  "connId": 2147483706,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:142",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "sessionId": "cmfqysatgznnbkkig5n9v78q8",
  "terminatorAddress": "5B2U1sBaDiWVOjLGoPaOE7",
  "time": "2025-09-19T15:05:51.161Z"
}
{
  "_context": "{c/5OiIr2uYNf35iCLMvlz21a|@/ZEPlFN28jtSWEAXI0J43b}<Terminator>",
  "circuitId": "5OiIr2uYNf35iCLMvlz21a",
  "error": "cannot forward payload, no forward table for circuit=5OiIr2uYNf35iCLMvlz21a src=ZEPlFN28jtSWEAXI0J43b",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 0,
  "time": "2025-09-19T15:05:56.162Z"
}
{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfqysannznn9kkig0edtiqzr",
  "attempt": 0,
  "attemptNumber": "1",
  "binding": "edge",
  "circuitId": "5OiIr2uYNf35iCLMvlz21a",
  "context": "ch{ctrl}->u{reconnecting}->i{dc/JO5x}",
  "destination": "5B2U1sBaDiWVOjLGoPaOE7",
  "error": "error creating route for [c/5OiIr2uYNf35iCLMvlz21a]: timeout waiting for message reply: context deadline exceeded",
  "file": "github.com/openziti/ziti/router/handler_ctrl/route.go:140",
  "func": "github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail",
  "level": "error",
  "msg": "failure while handling route update",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "sessionId": "cmfqysatgznnbkkig5n9v78q8",
  "time": "2025-09-19T15:05:56.162Z"
}
{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfqysannznn9kkig0edtiqzr",
  "attemptNumber": "2",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "5OiIr2uYNf35iCLMvlz21a",
  "connId": 2147483707,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:142",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "sessionId": "cmfqysatgznnbkkig5n9v78q8",
  "terminatorAddress": "5B2U1sBaDiWVOjLGoPaOE7",
  "time": "2025-09-19T15:05:56.202Z"
}
{
  "_context": "{c/2dapZPkNugqIetRNcC9VcP|@/4adT69wJmyaTfGXQKZKd8R}<Terminator>",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/link_send_buffer.go:296",
  "func": "github.com/openziti/sdk-golang/xgress.(*LinkSendBuffer).run",
  "level": "warning",
  "msg": "closing while buffer contains unacked payloads",
  "payloadCount": 59,
  "time": "2025-09-19T15:05:57.502Z"
}
{
  "_context": "{c/2dapZPkNugqIetRNcC9VcP|@/4adT69wJmyaTfGXQKZKd8R}<Terminator>",
  "circuitId": "2dapZPkNugqIetRNcC9VcP",
  "error": "cannot forward payload, no forward table for circuit=2dapZPkNugqIetRNcC9VcP src=4adT69wJmyaTfGXQKZKd8R",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 1334,
  "time": "2025-09-19T15:05:57.502Z"
}
{
  "_context": "{c/2dapZPkNugqIetRNcC9VcP|@/4adT69wJmyaTfGXQKZKd8R}<Terminator>",
  "error": "write closed",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/xgress.go:870",
  "func": "github.com/openziti/sdk-golang/xgress.(*Xgress).forwardPayload",
  "level": "error",
  "msg": "failure to buffer payload",
  "time": "2025-09-19T15:05:57.502Z"
}
{
  "circuitCount": 2,
  "ctrlId": "dc",
  "file": "github.com/openziti/ziti/router/forwarder/faulter.go:107",
  "func": "github.com/openziti/ziti/router/forwarder.(*Faulter).run",
  "level": "warning",
  "msg": "reported forwarding faults",
  "time": "2025-09-19T15:06:00.725Z"
}
{
  "circuitId": "2dapZPkNugqIetRNcC9VcP",
  "file": "github.com/openziti/ziti/router/forwarder/forwarder.go:155",
  "func": "github.com/openziti/ziti/router/forwarder.(*Forwarder).Unroute",
  "level": "info",
  "msg": "circuit unrouted",
  "time": "2025-09-19T15:06:00.793Z"
}
{
  "circuitId": "5OiIr2uYNf35iCLMvlz21a",
  "file": "github.com/openziti/ziti/router/forwarder/forwarder.go:155",
  "func": "github.com/openziti/ziti/router/forwarder.(*Forwarder).Unroute",
  "level": "info",
  "msg": "circuit unrouted",
  "time": "2025-09-19T15:06:00.793Z"
}
{
  "_context": "{c/5OiIr2uYNf35iCLMvlz21a|@/4qrSUoDb500NrWVmVnCQkf}<Terminator>",
  "circuitId": "5OiIr2uYNf35iCLMvlz21a",
  "error": "cannot forward payload, no forward table for circuit=5OiIr2uYNf35iCLMvlz21a src=4qrSUoDb500NrWVmVnCQkf",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 0,
  "time": "2025-09-19T15:06:00.794Z"
}
13

This simple scenario does not work under 1.6.7. I remember It did when ziti 1.5.4 was installed.

curl --proxy socks5://127.0.0.1:9191 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso

Ctrl-C

curl -I --proxy socks5://127.0.0.1:9191 https://www.google.com:443
curl: (97) Recv failure: Connection reset by peer

chain: curl <> zrok <20Mbps> router1 <100Mbps> router2 <> zrok

14

Hello,
Thank you for sharing the test case. Can you also post the zrok commands you're using to set up the share, please?

Thank you,
Paul

15 
zrok share private  -b socks --headless
[   1.480]    INFO sdk-golang/ziti.(*listenerManager).createSessionWithBackoff: {session token=[]} new service session
[   1.619]    INFO main.(*sharePrivateCommand).shareLocal: allow other to access your share with the following command:
zrok access private qe5mnyxgedym

On your PC you run

zrok access private qe5mnyxgedym -b 127.0.0.1:4141 --headless

Test:

curl --proxy socks5://127.0.0.1:4141 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso

Ctrl-C

curl -I --proxy socks5://127.0.0.1:4141 https://www.google.com:443
curl: (97) Recv failure: Connection reset by peer

zrok v1.1.2, ziti v1.6.7 (all routers and one HA controller)

16

Thank you, I'll see if I can reproduce this locally.

17

It is always the same scenario.

  1. the router has been set in error state
{
  "channelId": "ch{l/4noydwoTYeEXSYmkbFOmyx}->u{classic}->i{4noydwoTYeEXSYmkbFOmyx/0979ada5-42ba-489e-877d-004798571316}",
  "error": "timeout waiting to put message in send queue (context deadline exceeded)",
  "file": "github.com/openziti/channel/v4@v4.2.21/heartbeater.go:189",
  "func": "github.com/openziti/channel/v4.(*heartbeater).sendHeartbeatIfQueueFree",
  "level": "error",
  "msg": "handleUnresponded failed to send heartbeat",
  "time": "2025-09-20T13:57:14.250Z"
}
{
  "RTT": 6486,
  "_context": "{c/2Aamu0RSREFn8ZfYwF9gZl|@/3JjDhsR6Davg3KfTkNPYZG}<Terminator>",
  "circuitId": "2Aamu0RSREFn8ZfYwF9gZl",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/link_send_buffer.go:163",
  "func": "github.com/openziti/sdk-golang/xgress.(*LinkSendBuffer).ReceiveAcknowledgement",
  "level": "error",
  "linkRecvBufferSize": 32785,
  "msg": "payload buffer closed",
  "seq": "[117]",
  "time": "2025-09-20T13:59:21.764Z"
}
{
  "_context": "{c/117e7O985cDdIaeNaRCWTm|@/19Q5PtL85Cj2cUhtwECtxK}<Terminator>",
  "error": "write closed",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/xgress.go:870",
  "func": "github.com/openziti/sdk-golang/xgress.(*Xgress).forwardPayload",
  "level": "error",
  "msg": "failure to buffer payload",
  "time": "2025-09-20T13:59:33.261Z"
}
{
  "_context": "{c/2TX1iiV9FbKD75IPbnISJw|@/7mUYiZwnA0Pl31VlL8ESQK}<Terminator>",
  "circuitId": "2TX1iiV9FbKD75IPbnISJw",
  "error": "cannot forward payload, no forward table for circuit=2TX1iiV9FbKD75IPbnISJw src=7mUYiZwnA0Pl31VlL8ESQK",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 0,
  "time": "2025-09-20T13:59:49.037Z"
}
  1. Then It is impossible to dial the service. The router reports errors:
{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfs5fj3g1i28vdigrhdddqc9",
  "attempt": 0,
  "attemptNumber": "1",
  "binding": "edge",
  "circuitId": "2TX1iiV9FbKD75IPbnISJw",
  "context": "ch{ctrl}->u{reconnecting}->i{dc/zDyY}",
  "destination": "7T7EfU0OBXLkURS9NPDn4O",
  "error": "error creating route for [c/2TX1iiV9FbKD75IPbnISJw]: timeout waiting for message reply: context deadline exceeded",
  "file": "github.com/openziti/ziti/router/handler_ctrl/route.go:140",
  "func": "github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail",
  "level": "error",
  "msg": "failure while handling route update",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "sessionId": "cmfs5fj9t1i2avdigu6o0ch3d",
  "time": "2025-09-20T13:59:49.038Z"
}
{
  "_context": "{c/7g8fzmgdvqzTYSBCZQvwLn|@/2L0PyeoVFjqbeo5VBx7AEQ}<Terminator>",
  "circuitId": "7g8fzmgdvqzTYSBCZQvwLn",
  "error": "cannot forward payload, no forward table for circuit=7g8fzmgdvqzTYSBCZQvwLn src=2L0PyeoVFjqbeo5VBx7AEQ",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 0,
  "time": "2025-09-20T13:59:54.039Z"
}

In the ziti-controller’s log we can see

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfs5fj3g1i28vdigrhdddqc9",
  "attemptNumber": 2,
  "circuitId": "2TX1iiV9FbKD75IPbnISJw",
  "file": "github.com/openziti/ziti/controller/network/routesender.go:197",
  "func": "github.com/openziti/ziti/controller/network.(*routeSender).handleRouteSend",
  "level": "warning",
  "msg": "received failed route status from [r/JoOo6WCyhW] for attempt [#1] of [s/2TX1iiV9FbKD75IPbnISJw] (error creating route for [c/2TX1iiV9FbKD75IPbnISJw]: timeout waiting for message reply: context deadline exceeded)",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "sessionId": "cmfs5fj9t1i2avdigu6o0ch3d",
  "time": "2025-09-20T13:59:59.004Z"
}
{
  "_channels": [
    "selectPath"
  ],
  "apiSessionId": "cmfs5fj3g1i28vdigrhdddqc9",
  "attemptNumber": 2,
  "circuitId": "2TX1iiV9FbKD75IPbnISJw",
  "error": "error creating route for [s/2TX1iiV9FbKD75IPbnISJw] on [r/JoOo6WCyhW] (error creating route for [c/2TX1iiV9FbKD75IPbnISJw]: timeout waiting for message reply: context deadline exceeded)",
  "file": "github.com/openziti/ziti/controller/network/network.go:663",
  "func": "github.com/openziti/ziti/controller/network.(*Network).CreateCircuit",
  "level": "warning",
  "msg": "route attempt for circuit failed",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "serviceName": "XXXX",
  "sessionId": "cmfs5fj9t1i2avdigu6o0ch3d",
  "time": "2025-09-20T13:59:59.004Z"
}
18

I ve done test with only one router 1.6.7

zrok share private  -b socks --headless
[   0.877]    INFO sdk-golang/ziti.(*listenerManager).createSessionWithBackoff: {session token=[ccc293dc-4db8-40bf-b8f7-039d008eb0b7]} new service session
[   0.933]    INFO main.(*sharePrivateCommand).shareLocal: allow other to access your share with the following command:
zrok access private iun3mu62c18u

on PC you setup the binding

zrok access private iun3mu62c18u -b 127.0.0.1:4141 --headless

Then you run the test:

curl --proxy socks5://127.0.0.1:4141 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso

Ctrl-C

The router’s log contains errors

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfu2cbhn3zomvdigs4x0pyum",
  "attemptNumber": "1",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "1cQMbYwtPGdO0UbzFKPC1m",
  "connId": 2147483648,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:142",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "3eJoLyVB22BICWJRjPpoBY",
  "sessionId": "cmfu2cbng3zoqvdigftkkt7ha",
  "terminatorAddress": "2aDbDnILYWE2bPVZqe0hWo",
  "time": "2025-09-21T19:01:04.841Z"
}
{
  "_context": "{c/1cQMbYwtPGdO0UbzFKPC1m|@/6hSjSRXYhY8Lv2HRFo6mTW}<Terminator>",
  "circuitId": "1cQMbYwtPGdO0UbzFKPC1m",
  "error": "cannot forward payload, no destination for circuit=1cQMbYwtPGdO0UbzFKPC1m src=6hSjSRXYhY8Lv2HRFo6mTW dst=5NzBXZWT5IFfzUgqjREYtF",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 3766,
  "time": "2025-09-21T19:01:43.063Z"
}
{
  "_context": "{c/1cQMbYwtPGdO0UbzFKPC1m|@/6hSjSRXYhY8Lv2HRFo6mTW}<Terminator>",
  "error": "write closed",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/xgress.go:870",
  "func": "github.com/openziti/sdk-golang/xgress.(*Xgress).forwardPayload",
  "level": "error",
  "msg": "failure to buffer payload",
  "time": "2025-09-21T19:01:43.063Z"
}
{
  "_context": "{c/1cQMbYwtPGdO0UbzFKPC1m|@/6hSjSRXYhY8Lv2HRFo6mTW}<Terminator>",
  "file": "github.com/openziti/sdk-golang@v1.2.2/xgress/link_send_buffer.go:296",
  "func": "github.com/openziti/sdk-golang/xgress.(*LinkSendBuffer).run",
  "level": "warning",
  "msg": "closing while buffer contains unacked payloads",
  "payloadCount": 2,
  "time": "2025-09-21T19:01:43.063Z"
}
{
  "circuitCount": 1,
  "ctrlId": "dc",
  "file": "github.com/openziti/ziti/router/forwarder/faulter.go:107",
  "func": "github.com/openziti/ziti/router/forwarder.(*Faulter).run",
  "level": "warning",
  "msg": "reported forwarding faults",
  "time": "2025-09-21T19:01:43.613Z"
}
{
  "circuitId": "1cQMbYwtPGdO0UbzFKPC1m",
  "file": "github.com/openziti/ziti/router/forwarder/forwarder.go:155",
  "func": "github.com/openziti/ziti/router/forwarder.(*Forwarder).Unroute",
  "level": "info",
  "msg": "circuit unrouted",
  "time": "2025-09-21T19:01:43.633Z"
}

The service does not reply.

curl -4 -I --proxy socks5://127.0.0.1:4141 https://www.google.com:443
curl: (97) Recv failure: Connection reset by peer

In the router’s log you find

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfu2cbhn3zomvdigs4x0pyum",
  "attemptNumber": "1",
  "bindConnId": 1,
  "binding": "edge",
  "circuitId": "5S4Eq0PpfEweOGqOu6ZFJO",
  "connId": 2147483649,
  "file": "github.com/openziti/ziti/router/xgress_edge/dialer.go:142",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*dialer).Dial",
  "level": "info",
  "msg": "sending dial request to sdk",
  "serviceId": "3eJoLyVB22BICWJRjPpoBY",
  "sessionId": "cmfu2cbng3zoqvdigftkkt7ha",
  "terminatorAddress": "2aDbDnILYWE2bPVZqe0hWo",
  "time": "2025-09-21T19:02:59.002Z"
}
{
  "_context": "{c/5S4Eq0PpfEweOGqOu6ZFJO|@/deH7a9AvFhyaMmLJLwTSl}<Terminator>",
  "circuitId": "5S4Eq0PpfEweOGqOu6ZFJO",
  "error": "cannot forward payload, no forward table for circuit=5S4Eq0PpfEweOGqOu6ZFJO src=deH7a9AvFhyaMmLJLwTSl",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 1,
  "seq": 0,
  "time": "2025-09-21T19:03:04.003Z"
}

There are frequent failures. My network does not work properly.

19

If 1.6.7 is not working for you, I'd recommend downgrading to 1.5.4 until the issue is sorted out. I'll be working on replicating this starting tomorrow.

20

Ok, I got this set up today.

1 OpenZiti Controller running OpenZiti v1.6.7 in HA mode
1 OpenZiti Edge Router running OpenZiti v1.6.7
1 zrok controller running v1.1.2

Run the share

$ zrok share private  -b socks --headless
[   0.144]    INFO sdk-golang/ziti.(*listenerManager).createSessionWithBackoff: {session token=[1ce9b918-1328-477f-a8f6-542f3971e848]} new service session
[   0.177]    INFO main.(*sharePrivateCommand).shareLocal: allow other to access your share with the following command:
zrok access private olq7bt6z3s7q
[  29.858]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 134.158.69.171:443
[  39.901]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 142.251.40.132:443
[  41.076]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 142.251.40.132:443
[  42.924]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 134.158.69.171:443
[  69.804]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 134.158.69.171:443
[  76.519]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 142.251.40.132:443
[  77.469]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 142.251.40.132:443
[  78.139]    INFO main.(*sharePrivateCommand).shareLocal:  -> CONNECT 142.251.40.132:443

Run the share access

$ zrok access private olq7bt6z3s7q -b 127.0.0.1:4141 --headless
[   0.108]    INFO main.(*accessPrivateCommand).accessLocal: access the zrok share at the following endpoint: http://127.0.0.1:4141
[  16.630]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:48166 -> ACCEPT olq7bt6z3s7q
[  26.669]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:60456 -> ACCEPT olq7bt6z3s7q
[  27.848]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:60470 -> ACCEPT olq7bt6z3s7q
[  29.696]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:60480 -> ACCEPT olq7bt6z3s7q
[  56.576]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:48202 -> ACCEPT olq7bt6z3s7q
[  63.293]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:48214 -> ACCEPT olq7bt6z3s7q
[  64.243]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:48216 -> ACCEPT olq7bt6z3s7q
[  64.911]    INFO main.(*accessPrivateCommand).accessLocal: 127.0.0.1:48222 -> ACCEPT olq7bt6z3s7q

Running curls:

$ curl --proxy socks5://127.0.0.1:4141 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 2287M    0 6512k    0     0  1609k      0  0:24:15  0:00:04  0:24:11 1609k^C
$ curl --proxy socks5://127.0.0.1:4141 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2 2287M    2 55.9M    0     0  10.8M      0  0:03:30  0:00:05  0:03:25 11.2M^C
$ curl -4 -I --proxy socks5://127.0.0.1:4141 https://www.google.com:443
HTTP/2 200 
content-type: text/html; charset=ISO-8859-1
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-dlJ1JBPxlPh1ESnRF5XbGw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
accept-ch: Sec-CH-Prefers-Color-Scheme
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
date: Tue, 23 Sep 2025 18:23:03 GMT
server: gws
x-xss-protection: 0
x-frame-options: SAMEORIGIN
expires: Tue, 23 Sep 2025 18:23:03 GMT
cache-control: private
set-cookie: AEC=AaJma5vyeolu58kWaCceBP2f45Yr9gCkggaCn9h9GqYMkpBM5D-uK4Gqumk; expires=Sun, 22-Mar-2026 18:23:03 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: NID=525=Mh8BD5Qw9H7w14XcByLsZ5qQT2JwahlqLo6XTgPvFkI5t39JcezX3pIT0CMDmqD3moKa67ZBq3FGINlOl_SW-6P5wn1gHsTi5EV2awOoBKY-GS711fFIIYXnnLDVXumWbXZTxmzaaQmUFYAt9-bURxSo56YgWCM5lVfar5jcFNeuUvUjMbnU1pHQ2jQioUOfeyZKD_Y7gzW-SFcKfCu1; expires=Wed, 25-Mar-2026 18:23:03 GMT; path=/; domain=.google.com; HttpOnly
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

$ curl -4 -I --proxy socks5://127.0.0.1:4141 https://www.google.com:443
HTTP/2 200 
... rest stripped out for brevity, but same as above...

$ curl -4 -I --proxy socks5://127.0.0.1:4141 https://www.google.com:443
HTTP/2 200 
... rest stripped out for brevity, same as above...

$ curl --proxy socks5://127.0.0.1:4141 -o /dev/null https://mirror.in2p3.fr/pub/fedora/linux/releases/42/Workstation/x86_64/iso/Fedora-Workstation-Live-42-1.1.x86_64.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 2287M    0 19.3M    0     0  2075k      0  0:18:48  0:00:09  0:18:39 2550k^C
$ curl -4 -I --proxy socks5://127.0.0.1:4141 https://www.google.com:443
HTTP/2 200 
... rest stripped out for brevity, same as above...

$

I didn't hit the issue you're reporting. There must be some difference in our setups.

As I was going back through the logs you posted, one thing that stuck out was this message:

{
  "_channels": [
    "establishPath"
  ],
  "apiSessionId": "cmfs5fj3g1i28vdigrhdddqc9",
  "attempt": 0,
  "attemptNumber": "1",
  "binding": "edge",
  "circuitId": "2TX1iiV9FbKD75IPbnISJw",
  "context": "ch{ctrl}->u{reconnecting}->i{dc/zDyY}",
  "destination": "7T7EfU0OBXLkURS9NPDn4O",
  "error": "error creating route for [c/2TX1iiV9FbKD75IPbnISJw]: timeout waiting for message reply: context deadline exceeded",
  "file": "github.com/openziti/ziti/router/handler_ctrl/route.go:140",
  "func": "github.com/openziti/ziti/router/handler_ctrl.(*routeHandler).fail",
  "level": "error",
  "msg": "failure while handling route update",
  "serviceId": "Aus1zAXp334UUd1QCtp8Y",
  "sessionId": "cmfs5fj9t1i2avdigu6o0ch3d",
  "time": "2025-09-20T13:59:49.038Z"
}

That would lead me to believe that the zrok process hosting the share (the one running zrok share private is either deadlocked or too busy.

Could you try grabbing a stack dump from the hosting process? You can run kill -3 <pid of process>.

That will hopefully reveal any thread contention in the sdk or the zrok hosting code. If that doesn't show anything, we'll have to drill in further and see if we can isolate the difference in setup.

Thank you,
Paul