Why I have "reported forwarding faults"?

zrok
1

Since the very beginning I have strange notifications in the system's log. This produces only on the host where the ziti controller is installed.
The second ziti router on a different host does not print the message.

router: "reported forwarding faults"

Apr 23 14:20:57  ziti[3024207]: {"circuitId":"a834-RJ2b","ctrlId":"NetFoundry Inc. Client u98mGBfk5","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":160841000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2025-04-23T14:20:57.157Z"}
Apr 23 14:20:57  ziti[3024207]: {"circuitId":"3EMV-RJ5b","ctrlId":"NetFoundry Inc. Client u98mGBfk5","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":153720000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2025-04-23T14:20:57.157Z"}
Apr 23 14:20:57  ziti[3024207]: {"circuitId":"r4eOqOJ5b","ctrlId":"NetFoundry Inc. Client u98mGBfk5","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":255030000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2025-04-23T14:20:57.157Z"}
Apr 23 14:20:57  ziti[3024207]: {"circuitId":"wvbR-Rs5b","ctrlId":"NetFoundry Inc. Client u98mGBfk5","file":"github.com/openziti/ziti/router/forwarder/scanner.go:85","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","idleThreshold":60000000000,"idleTime":322140000000,"level":"warning","msg":"circuit exceeds idle threshold","time":"2025-04-23T14:20:57.158Z"}
Apr 23 14:20:57  ziti[3024207]: {"circuitCount":4,"ctrlId":"NetFoundry Inc. Client u98mGBfk5","file":"github.com/openziti/ziti/router/forwarder/scanner.go:100","func":"github.com/openziti/ziti/router/forwarder.(*Scanner).scan","level":"warning","msg":"sent confirmation for circuits","time":"2025-04-23T14:20:57.158Z"}
Apr 23 14:20:57  ziti[3024207]: {"circuitCount":1,"ctrlId":"NetFoundry Inc. Client u98mGBfk5","file":"github.com/openziti/ziti/router/forwarder/faulter.go:107","func":"github.com/openziti/ziti/router/forwarder.(*Faulter).run","level":"warning","msg":"reported forwarding faults","time":"2025-04-23T14:20:57.199Z"}

controller: "sent unroute for circuit to router in response to forwarding fault"

Apr 23 14:24:27  ziti[3024098]: {"file":"github.com/openziti/ziti/controller/network/fault.go:32","func":"github.com/openziti/ziti/controller/network.(*Network).fault","level":"info","msg":"network fault processing for [1] circuits","time":"2025-04-23T14:24:27.207Z"}
Apr 23 14:24:27  ziti[3024098]: {"circuitId":"eRvk2Rs5n","file":"github.com/openziti/ziti/controller/network/fault.go:49","func":"github.com/openziti/ziti/controller/network.(*Network).fault","level":"info","msg":"sent unroute for circuit to router in response to forwarding fault","routerId":"NzPxsJsTC","time":"2025-04-23T14:24:27.208Z"}
Apr 23 14:24:42  ziti[3024098]: {"file":"github.com/openziti/ziti/controller/network/fault.go:32","func":"github.com/openziti/ziti/controller/network.(*Network).fault","level":"info","msg":"network fault processing for [1] circuits","time":"2025-04-23T14:24:42.208Z"}
Apr 23 14:24:42  ziti[3024098]: {"circuitId":"j4z0DRJ2b","file":"github.com/openziti/ziti/controller/network/fault.go:49","func":"github.com/openziti/ziti/controller/network.(*Network).fault","level":"info","msg":"sent unroute for circuit to router in response to forwarding fault","routerId":"NzPxsJsTC","time":"2025-04-23T14:24:42.208Z"}


journalctl --since "4 hours ago" -u ziti-router.service -g 'reported forwarding faults'  | wc -l
75
2

I'm unsure of the precise meaning of this informational message, and will provide some additional context to begin shedding light on the cause. I assume Ziti is functioning, and you're concerned the message could indicate a misconfiguration or malfunction.

Here's an explanation of this informational message from the reported code location: ziti/controller/network/fault.go at v1.5.4 · openziti/ziti · GitHub

If the owner is unknown, we can't unroute because the circuit may be owned by some other controller
UnknownOwner faults come from a link forwarding operation and are generally caused by a missing forward
table entry. In that case there's nothing to unroute anyway.

This is from a commit added two years ago: Notify controller of forward faults on links. Fixes #724. If reroute … · openziti/ziti@ef555a5 · GitHub with message:

Notify controller of forward faults on links. Fixes #724. If reroute fails, circuit should be torn down. Fixes openziti/fabric#725