Cannot create Circuit after Controller reboot

Support
1

Hi Everyone,

after my last problem with the missing terminators after the controller reboot, I now have another one, which probably already existed last time.

As soon as I restart the controller, I get the following error messages on all routers:

Jun 06 10:04:21 edge1 ziti[817]: {"ctrlId":"NetFoundry Inc. Client xprQlJpGf","file":"github/openziti/ziti/router/state/manager.go:256","func":"github/openziti/ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription","level":"info","msg":"no current data model subscription active, subscribing","prevCtrlId":"NetFoundryInc. Client xprQlJpGf","time":"2025-06-06T10:04:21.594Z"}
Jun 06 10:04:21 edge1 ziti[817]: {"ctrlId":"NetFoundry Inc. Client xprQlJpGf","currentIndex":110,"file":"github/openziti/ziti/router/state/manager.go:306","func":"github/openziti/ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates","level":"info","msg":"subscribed to new controller for router data model changes","renew":true,"time":"2025-06-06T10:04:21.594Z"}
Jun 06 10:04:21 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":270,"connId":141,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:21.612Z","token":"3b384ae0-5947-4bd7-9103-ce52e8b2130b","type":"EdgeConnectType"}
Jun 06 10:04:22 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":272,"connId":142,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:22.130Z","token":"3b384ae0-5947-4bd7-9103-ce52e8b2130b","type":"EdgeConnectType"}
Jun 06 10:04:45 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":274,"connId":143,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:45.218Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:45 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":276,"connId":144,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:45.477Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:45 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":278,"connId":145,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:45.742Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:45 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":280,"connId":146,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:45.994Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:46 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":282,"connId":147,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:46.258Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:46 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":284,"connId":148,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:46.506Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:46 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":286,"connId":149,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:46.771Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:47 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":288,"connId":150,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:47.021Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:47 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":290,"connId":151,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:47.287Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}
Jun 06 10:04:47 edge1 ziti[817]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{ziti-sdk-c[2]@T14/z5Mz}","chSeq":292,"connId":152,"edgeSeq":0,"file":"github/openziti/ziti/router/xgress_edge/listener.go:143","func":"github/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"error","msg":"no controller available, cannot create circuit","time":"2025-06-06T10:04:47.534Z","token":"92fe0e4f-b19d-4a00-a1e8-f7d142e18803","type":"EdgeConnectType"}

I have the following error message on the controller:

Jun 06 10:03:51 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"bwDLT6Mwv","time":"2025-06-06T10:03:51.599Z","timelineId":"-cctKnaHR"}
Jun 06 10:03:51 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"bwDLT6Mwv","routerName":"edge1.ziti.example.com ","subscriptionDuration":"2025-06-06 10:08:51.599120962 +0000 UTC m=+534.180820271","time":"2025-06-06T10:03:51.599Z","timelineId":"-cctKnaHR"}
Jun 06 10:03:56 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"lQhjli5OW","time":"2025-06-06T10:03:56.507Z","timelineId":"-cctKnaHR"}
Jun 06 10:03:56 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"lQhjli5OW","routerName":"edge2.ziti.example.com ","subscriptionDuration":"2025-06-06 10:08:56.507309491 +0000 UTC m=+539.089008901","time":"2025-06-06T10:03:56.507Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:02 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"jcaxHi0Lb-","time":"2025-06-06T10:04:02.727Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:02 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"jcaxHi0Lb-","routerName":"private2.ziti.example.com ","subscriptionDuration":"2025-06-06 10:09:02.727214734 +0000 UTC m=+545.308914074","time":"2025-06-06T10:04:02.727Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:20 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"gWg1W6qwvF","time":"2025-06-06T10:04:20.156Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:20 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"gWg1W6qwvF","routerName":"private1.ziti.example.com ","subscriptionDuration":"2025-06-06 10:09:20.156558174 +0000 UTC m=+562.738257701","time":"2025-06-06T10:04:20.156Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:21 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"bwDLT6Mwv","time":"2025-06-06T10:04:21.598Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:21 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"bwDLT6Mwv","routerName":"edge1.ziti.example.com ","subscriptionDuration":"2025-06-06 10:09:21.598771409 +0000 UTC m=+564.180470811","time":"2025-06-06T10:04:21.598Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:26 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"lQhjli5OW","time":"2025-06-06T10:04:26.506Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:26 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"lQhjli5OW","routerName":"edge2.ziti.example.com ","subscriptionDuration":"2025-06-06 10:09:26.506461098 +0000 UTC m=+569.088160421","time":"2025-06-06T10:04:26.506Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:32 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"jcaxHi0Lb-","time":"2025-06-06T10:04:32.727Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:32 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"jcaxHi0Lb-","routerName":"private2.ziti.example.com ","subscriptionDuration":"2025-06-06 10:09:32.727154933 +0000 UTC m=+575.308854311","time":"2025-06-06T10:04:32.727Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:50 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"gWg1W6qwvF","time":"2025-06-06T10:04:50.156Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:50 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"gWg1W6qwvF","routerName":"private1.ziti.example.com ","subscriptionDuration":"2025-06-06 10:09:50.156554011 +0000 UTC m=+592.738253441","time":"2025-06-06T10:04:50.156Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:51 ctrl1 ziti[1135]: {"file":"github/openziti/ziti/controller/handler_edge_ctrl/subscribe_to_data_model.go:54","func":"github/openziti/ziti/controller/handler_edge_ctrl.(*subscribeToDataModelHandler).HandleReceive","index":110,"level":"info","msg":"data model subscription request received","routerId":"bwDLT6Mwv","time":"2025-06-06T10:04:51.598Z","timelineId":"-cctKnaHR"}
Jun 06 10:04:51 ctrl1 ziti[1135]: {"currentIndex":110,"file":"github/openziti/ziti/controller/sync_strats/rtx.go:187","func":"github/openziti/ziti/controller/sync_strats.(*RouterSender).handleSyncRequest","level":"info","msg":"data model subscription started","renew":true,"requestedIndex":110,"routerId":"bwDLT6Mwv","routerName":"edge1.ziti.example.com ","subscriptionDuration":"2025-06-06 10:09:51.598412676 +0000 UTC m=+594.180112033","time":"2025-06-06T10:04:51.598Z","timelineId":"-cctKnaHR"}

After I also restarted all routers, everything worked again.

I have the following setup: One standalone controller v1.5.4 and 4 routers also v1.5.4, 2 edge route and 2 private/internal routers.