Failed to send update router interfaces request to controller

Ziti Overlay
1

I have tons of messages on routers

Aug 18 20:50:21  ziti[767]: {"error":"timeout waiting for message reply: context deadline exceeded","file":"github.com/openziti/ziti/router/interfaces/interfaces.go:155","func":"github.com/openziti/ziti/router/interfaces.(*InterfaceReporter).reportUpdatedInterfaces","level":"error","msg":"failed to send update router interfaces request to controller","time":"2025-08-18T20:50:21.655Z"}

routers : ziti version v1.6.7

ziti controller: ziti version v1.5.4

2

The release notes tell you how to disable the reporting, see below. I may add an auto-disable if the controller version is below 1.6.3.

Release 1.6.3

What's New

  • Router Network Interface Discovery

Router Interface Discovery

Routers can now discover their network interfaces and publish this information to the
controller.

This feature will be used in the future to allow controller side configuration of
router link listeners and edge listeners.

Update Router Configuration

There is new router configuration to manage this:

interfaceDiscovery:
  # This feature is enabled by default, but can be disabled by setting this to true
  disabled: false

  # How often to poll for interface changes. Defaults to 1 minute
  checkInterval: 1m

  # How often to report the current set of interfaces, when nothing has changed.
  # This is a failsafe reporting mechanism, in the very unlikely event that an 
  # earlier change report was lost or disregarded due to distributed controller
  # eventual consistency
  minReportInterval: 24h
3

Thank you,

With disabled: true these error messages go away.

4

There is still a problem.

Aug 18 21:39:57 ziti[78442]: {"_context":"{c/nkmYQk3Rm|@/3wNLQ}\u003cTerminator\u003e","circuitId":"nkmYQk3Rm","error":"cannot forward payload, no forward table for circuit=nkmYQk3Rm src=3wNLQ","file":"github.com/openziti/ziti/router/handler_xgress/data_plane.go:58","func":"github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload","level":"error","msg":"unable to forward payload","origin":1,"seq":21,"time":"2025-08-18T21:39:57.932Z"}

ziti 1.5.4 did not have this error.

5

It's very hard to tell if this actually a problem. It can happen as part of normal circuit teardown, depending on timing, although I've been working on trying to minimize these. It can also happen if external clients don't close connections cleanly.

If you are seeing unexpected circuit failures, it may be a problem, otherwise you can ignore those errors.

6 
ziti fabric list circuits 'limit none' -j | jq '.'


      "createdAt": "2025-08-19T14:03:54.668Z",
      "id": "3U.rg08FG",
      "tags": {
        "clientId": "ZId",
        "hostId": "ZZId",
        "serviceId": "1qckVqdrhmPI7JZoxNjQb2"
      },
      "updatedAt": "2025-08-19T14:03:54.668Z",
      "clientId": "cmeim1o3n01005iig6qmbfelx",
      "path": {
        "links": null,
        "nodes": [
          {
            "_links": {
              "self": {
                "href": "./routers/7H8BaaKm9F"
              },
              "terminators": {
                "href": "./routers/7H8BaaKm9F/terminators"
              }
            },
            "entity": "routers",
            "id": "RID",
            "name": "router"
          }
        ]
      },

There is proliferation of single node circuits like above (ziti controller 1.6.7) A path of this kind contains only one node.

I have never seen tadpoles on ziti controller 1.5.4.

7

With ziti 1.5.4 I have about 34 error every 12h on 6 routers:

      2 "payload buffer closed"
      3 "failed to dial fabric"
      3 "failure while handling route update"
      4 "terminator was replaced after being put into deleting state?!"
      6 "unable to forward payload"
     16 "failure to buffer payload"

With ziti 1.6.7 I have an error every 1 minute.

I should also add the following:

{
  "file": "github.com/openziti/ziti/router/xgress_edge/hosted.go:267",
  "func": "github.com/openziti/ziti/router/xgress_edge.(*hostedServiceRegistry).RemoveTerminatorsRateLimited.func1",
  "level": "error",
  "msg": "terminator was replaced after being put into deleting state?!",
  "terminatorId": "XD6xzbPRpygapI4l3TGSA",
  "time": "2025-08-19T14:17:55.472Z"
}

I am back to 1.5.4

8

Using 1.6.7 I have the same error every minute.

journalctl --since "3 minutes ago" -u ziti-router.service -g error | sed 's/.*{"/{"/' | jq '.'
{
  "_context": "{c/a5lYhWAaB|@/9Q4O}<Initiator>",
  "circuitId": "a5lYhWAaB",
  "error": "cannot forward payload, no forward table for circuit=a5lYhWAaB src=9Q4O",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 0,
  "seq": 16,
  "time": "2025-08-22T10:05:41.299Z"
}
{
  "_context": "{c/IZv6hhAaT|@/qO87}<Initiator>",
  "circuitId": "IZv6hhAaT",
  "error": "cannot forward payload, no forward table for circuit=IZv6hhAaT src=qO87",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 0,
  "seq": 16,
  "time": "2025-08-22T10:05:42.288Z"
}
{
  "_context": "{c/92FylhA2T|@/wg0v}<Initiator>",
  "circuitId": "92FylhA2T",
  "error": "cannot forward payload, no forward table for circuit=92FylhA2T src=wg0v",
  "file": "github.com/openziti/ziti/router/handler_xgress/data_plane.go:58",
  "func": "github.com/openziti/ziti/router/handler_xgress.(*dataPlaneAdapter).ForwardPayload",
  "level": "error",
  "msg": "unable to forward payload",
  "origin": 0,
  "seq": 20,
  "time": "2025-08-22T10:07:38.756Z"
}

What is wrong with my router?

Can ziti router 1.6.7 work with ziti-controller 1.5.4. It seems that the router’s circuits are destroyed so quickly.