Yes, there are three rate limiters to consider:
Auth Rate Limter
I agree with @TheLumberjack that this is likely what you are hitting.
This rate limiter exists because without it the system can enter a state where authentications get queued and take a long time to complete. The SDK then times out, waiting for the authentication and retries. The controller completes the auth, but the SDK no longer cares and the work is wasted. The rate limiter tries to ensure that we don't accept more auths than we can process before the SDK gives up on it. It can generally tell when we've hit the limit because writing the response back will fail, as the SDK has closed the connection.
You can try disabling it or bumping the max size if you think it will be helpful.
Also, we'll probably be able to remove this limiter in the future. Once users are using JWT-based authentication, which doesn't require storing anything in the DB, that will remove the primary bottleneck on auth. At that point, the rate limiter shouldn't be necessary anymore.
Example config was posted above.
Command Rate Limiter
Since model changes are serialized, we want to be sure that we're not building up a large queue of operations. If you're load testing using SDK clients, you shouldn't be hitting this. This primary comes into play when a lot of SDKs or routers are setting up terminators for hosting services at once. The routers handle this internally (including on behalf of SDKs setting up hosting), by retrying as appropriate when the controller is handling a flood of requests.
Example config
commandRateLimiter:
enabled: true
maxQueued: 100
TLS Handshake Limiter
In testing we found that TLS handshakes could use all the CPU on a controller, leaving little for other work to get done. This limiter is disabled by default for now.
Example config
tls:
handshakeTimeout: 30s
rateLimiter:
enabled: true
minSize: 5
maxSize: 250
Hope that's helpful, let me know if you've got any follow-up questions.
Paul