Router v2.0.0-pre10 and V2.0.0 stable deletes all terminators after ~ 12 minutes

Ziti Overlay
1

My issue is basically the same as router v2.0.0-pre6 deletes all terminators after ~ 12 minutes

The details are below but this was also the same for v2.0.0-pre10.

Environment

  • Controllers: OpenZiti v2.0.0 (apt -stable), 3-node HA cluster (RAFT), Debian 13 (trixie).
  • Routers: OpenZiti v2.0.0, 3 routers, all online, --tunneler-enabled.
  • Channel library: github.com/openziti/channel/v4@v4.3.11 (as logged by the router).
  • Hosting SDK application: zrok2 2.0.4 (dynamicProxyController service), sdk-golang v1.2.8 — but per the pre6 thread this also affects ziti-edge-tunnel / ziti-sdk-c, so it is not SDK-specific.

Full lifecycle (annotated router log, single terminator)

Terminator iRDhwFyRtdRixjaEBx6HH, service dynamicProxyController, on router-server3 (dn.QMaMs6), identity YDgqADM26.

Establish — works fine:

07:34:11  processBindV2: establishing terminator ... type EdgeBindType
07:34:11  evaluateEstablishQueue: queuing terminator to send create  state=1
07:34:11  establishTerminator: sending create terminator v2 request  ctrlId=ctrl3
07:34:11  evaluatePostCreateInspects: post-create inspect: sent inspect request  sendCount=1
07:34:11  channel/v4 impl.go:378  rx  WARNING  dropped message. type [60805], sequence [2], replyFor [2]
07:34:11  edgeTerminator.NotifyEstablished: sdk notified of terminator creation
07:34:11  edgeTerminator.updateState  newState=2 oldState=1  reason="create notification received"
07:34:11  markEstablishedEvent.handle  "terminator established"   <-- terminator is UP
07:34:27  validateTerminatorsV2: validating terminator  (controller validation OK)

Inspect loop — SDK replies, router drops every reply: each post-create inspect: sent inspect request (sendCount 1→12) is immediately followed by:

channel/v4 impl.go:378  rx  WARNING  dropped message. type [60805], sequence [N], replyFor [M]

Cadence: initial, +16s, +30s, then every 60s. The replyFor [M] values increment with each inspect but never match a waiter, so channelImpl.rx discards the InspectResponse.

Timeout and reap (false negative):

07:44:58  evaluatePostCreateInspects (hosted.go:622)  WARNING  "post-create inspect: timed out waiting for response, closing terminator"
07:44:58  edgeTerminator.setState  newState=3 oldState=2  reason="post-create inspect timed out"
07:45:58  evaluateDeleteQueue  "added terminator to batch delete"  state=3
07:45:58  RemoveTerminatorsRateLimited  "remove terminator succeeded"
07:45:58  Remove  "terminator removed from router set"  reason="controller delete success"

Creation 07:34:11 → reap 07:45:58 ≈ 11m47s (12 inspect attempts × ~60s).