The rate limiters discussed in this topic … How do they relate to the commandHandler.maxQueueSize parameter documented here ?
I often experience the following error when a flood of Terminator creation requests come in. Does that mean i’m hitting the commandHandler.maxQueueSize limit ?
{"_context":"ch{ctrl.mesh}-\u003eu{classic}-\u003ei{ctrl-2/X3r9}","error":"SERVER_TOO_MANY_REQUESTS: Too many requests to alter state have been issued. Please slow your request rate or try again later.","file":"github.com/openziti/ziti/controller/handler_peer_ctrl/command.go:80","func":"github.com/openziti/ziti/controller/handler_peer_ctrl.(*commandHandler).HandleReceive","level":"error","msg":"unable to queue command for processing","time":"2025-09-25T10:30:39.327Z"}
commandHandler.maxQueueSize is for the pool handling commands coming from other controllers. There's a pool so that we can offload incoming commands and not block the connection between controllers.
The command rate limiter on the other hand, takes effect on the leader and makes sure we can't have too many commands queued up.
Terminator creation works by sending a request to the controller. If the request times out or the controller is too busy, it will resend the request. It will also slow down the request rate if the controller indicates its too busy. The requests are idempotent, so if a create succeeds and then a new request comes in, we'll respond to the new request with success, without creating a duplicate terminator.
So the 'too many requests' error is letting you know that the system is temporarily not keeping up, but it shouldn't cause any problems, as the terminators should eventually get created.
Let me know if that makes sense or need any further clarification.
Paul