Hello everyone,
First of all, thank you for your work, you have put together a cool project.
I've been testing Openziti for our environment for a few weeks now.
I have noticed one big problem.
My setup looks like the following:
One controller, two edge routers and 2 private routers, all based on the “Prod” deployments, no quickstart.
All services are currently connected to the private Routers.
Everything works great until the controller is restarted.
All routers and tunnelers then reconnect, but no more terminators are created.
I get the following messages on the routers:
> Apr 30 12:39:14 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","file":"github.com/openziti/ziti/router/state/manager.go:256","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription","level":"info","msg":"no current data model subscription active, subscribing","prevCtrlId":"NetFoundry Inc. Client KbTdMRSoa","time":"2025-04-30T12:39:14.705Z"}
> Apr 30 12:39:14 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","currentIndex":121,"file":"github.com/openziti/ziti/router/state/manager.go:306","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates","level":"info","msg":"subscribed to new controller for router data model changes","renew":true,"time":"2025-04-30T12:39:14.705Z"}
> Apr 30 12:39:44 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","file":"github.com/openziti/ziti/router/state/manager.go:256","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription","level":"info","msg":"no current data model subscription active, subscribing","prevCtrlId":"NetFoundry Inc. Client KbTdMRSoa","time":"2025-04-30T12:39:44.706Z"}
> Apr 30 12:39:44 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","currentIndex":121,"file":"github.com/openziti/ziti/router/state/manager.go:306","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates","level":"info","msg":"subscribed to new controller for router data model changes","renew":true,"time":"2025-04-30T12:39:44.706Z"}
> Apr 30 12:40:14 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","file":"github.com/openziti/ziti/router/state/manager.go:256","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription","level":"info","msg":"no current data model subscription active, subscribing","prevCtrlId":"NetFoundry Inc. Client KbTdMRSoa","time":"2025-04-30T12:40:14.706Z"}
> Apr 30 12:40:14 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","currentIndex":121,"file":"github.com/openziti/ziti/router/state/manager.go:306","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates","level":"info","msg":"subscribed to new controller for router data model changes","renew":true,"time":"2025-04-30T12:40:14.706Z"}
> Apr 30 12:40:44 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","file":"github.com/openziti/ziti/router/state/manager.go:256","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription","level":"info","msg":"no current data model subscription active, subscribing","prevCtrlId":"NetFoundry Inc. Client KbTdMRSoa","time":"2025-04-30T12:40:44.706Z"}
> Apr 30 12:40:44 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","currentIndex":121,"file":"github.com/openziti/ziti/router/state/manager.go:306","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates","level":"info","msg":"subscribed to new controller for router data model changes","renew":true,"time":"2025-04-30T12:40:44.706Z"}
> Apr 30 12:41:14 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","file":"github.com/openziti/ziti/router/state/manager.go:256","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).checkRouterDataModelSubscription","level":"info","msg":"no current data model subscription active, subscribing","prevCtrlId":"NetFoundry Inc. Client KbTdMRSoa","time":"2025-04-30T12:41:14.706Z"}
> Apr 30 12:41:14 ziti-private-2 ziti[1563]: {"ctrlId":"NetFoundry Inc. Client KbTdMRSoa","currentIndex":121,"file":"github.com/openziti/ziti/router/state/manager.go:306","func":"github.com/openziti/ziti/router/state.(*ManagerImpl).subscribeToDataModelUpdates","level":"info","msg":"subscribed to new controller for router data model changes","renew":true,"time":"2025-04-30T12:41:14.706Z"}
I have to restart all the routers first, sometimes I even have to restart the individual tunnelers.
It gets really weird when I only restart the private routers, the terminators are created again. If I try to access a service, I see the following error message on the edge routers: No Controllers available, cannot create circuit (I could not reproduce this part, so no log extract), although I can see in the ZAC that the routers are connected.
The commands ziti edge policy-advisor services and ziti edge policy-advisor identities show no errors.
The error is not 100% reproducible, but I would say at least 50%.
Do you know the problem and is there perhaps a command that ensures that the routers “reboot”?
Thank you and best regards
Jonas