Hi *,
I have a ziti-controller and a ziti-router running in a k8s cluster. The router acts as tunneler to cluster-internal services. I exposed the controller and the edge listener via ingress controller.
I can then start ziti-edge-tunnel on my local machine and connect to the ziti fabric. I exposed some
cluster internal web services on the ziti overlay: the hello toy service and grafana. I can then issue curl http://hello.ziti locally and get the website back. So this works. It's also quite stable, I can hit the web site in a for loop on the shell 1000 times and it will deliver without problems. This also works in the browser. I can also access grafana, at least when I expose the port directly, foregoing ziti, without a problem. I can access grafana.ziti using curl via ziti, as long as I request one URL at a time: I can run wget -r http://grafana.ziti in a loop without issues.
Once i use the browser to access grafana.ziti, I get confronted with stalled connections, which end up being reported as "connection refused" by the browser, for files that would normally deliver. Some connections go through and deliver parts of the web page, but this will quickly end. At that point the ziti fabric becomes completely unresponsive, even for the plain curl requests to hello.ziti before. After a while this "normalizes" to the previous state. I have not changed any rate limiting options and used the provided Helm charts for deployment. Here are some excerpts from the controller logs captured during the browser requests:
{"file":"github.com/openziti/ziti/controller/network/fault.go:32","func":"github.com/openziti/ziti/controller/network.(*Network).fault","level":"info","msg":"network fault processing for [1] circuits","time":"2024-10-24T00:15:39.540Z"}
{"circuitId":"K8Y2dhISh","file":"github.com/openziti/ziti/controller/network/fault.go:49","func":"github.com/openziti/ziti/controller/network.(*Network).fault","level":"info","msg":"sent unroute for circuit to router in response to forwarding fault","routerId":"Nv4gLB6J0d","time":"2024-10-24T00:15:39.541Z"}
and the router
"_context":"{c/501Pzb5Sh|@/bqxn}\u003cTerminator\u003e","file":"github.com/openziti/ziti/router/xgress/xgress.go:575","func":"github.com/openziti/ziti/router/xgress.(*Xgress).flushSendThenClose.func1","level":"info","msg":"sending end of circuit payload","time":"2024-10-24T11:00:18.537Z"}
{"_context":"{c/501Pzb5Sh|@/4b8x}\u003cInitiator\u003e","circuitId":"501Pzb5Sh","file":"github.com/openziti/ziti/router/xgress/xgress.go:491","func":"github.com/openziti/ziti/router/xgress.(*Xgress).tx.func3","level":"info","msg":"payload sent [0 B]","origin":1,"seq":1,"time":"2024-10-24T11:00:18.538Z"}
{"_context":"{c/501Pzb5Sh|@/4b8x}\u003cInitiator\u003e","file":"github.com/openziti/ziti/router/xgress/xgress.go:575","func":"github.com/openziti/ziti/router/xgress.(*Xgress).flushSendThenClose.func1","level":"info","msg":"sending end of circuit payload","time":"2024-10-24T11:00:18.538Z"}
{"_context":"{c/501Pzb5Sh|@/4b8x}\u003cInitiator\u003e","circuitId":"501Pzb5Sh","error":"cannot forward payload, no destination for circuit=501Pzb5Sh src=4b8x dst=bqxn","file":"github.com/openziti/ziti/router/handler_xgress/receive.go:37","func":"github.com/openziti/ziti/router/handler_xgress.(*receiveHandler).HandleXgressReceive","level":"error","msg":"unable to forward payload","origin":0,"seq":3,"time":"2024-10-24T11:00:18.538Z"}
{"circuitCount":1,"ctrlId":"ziti-controller-ctrl-plane-identity","file":"github.com/openziti/ziti/router/forwarder/faulter.go:107","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run","level":"warning","msg":"reported forwarding faults","time":"2024-10-24T11:00:25.979Z"}
ziti fabric list circuits lists a number of circuits to the service. I suspect that the issue lies somehow with the amount of parallel connections that the browser opens. I am not sure how to proceed debugging this and would appreciate any pointers.
Thanks!
