I'm trying to understand what factors affect which Controller the ZET client chooses to connect to. I'd also like to know if there's anything that would cause the ZET client to move to a different controller other than the current controller becoming unavailable.
In my system i see client connections to controllers unevenly distributed. For example, roughly 50% of clients connect to Controller 1, 25% to Controller 2 and 25% to controller 3.
Is there a way to see the Controller that clients are connected to from the ziti cli ? Currently i'm grepping the ZET logs to get this information. For example i'd like to see how many clients are connected to each controller.
there is no really such thing as being connected to a controller.
SDK just needs to periodically hit controller API with REST requests. SDK may decide to switch to another controller for the next set of requests if current controller is offline, or after coming out of sleep.
The switch logic is non-deterministic. SDK is trying to randomize controller selection to avoid herding between different endpoints:
for example if ctrl1 went offline, we don't want all endpoints to switch ctrl2, we want to split them equally between active controllers.
This process may change in the future, but it general you can't infer anything from the current distribution. over a longer time period all controllers should be utilized equally especially once you have a larger client population.
The documentation mentions here that a HA controller could be placed geographically close to clients to reduce latency. But how would these clients know to use the geographically closer Controller rather than the others in the HA cluster ?
In my opinion, placement of controllers relative to SDK endpoints does not play a big role in latencies because SDK caches most of the data it receives from controller(s) and overlay connections are usually established without SDK contacting a controller.
However, there is an indirect effect:
Edge Router(s) placement is important to SDK endpoints, putting them closer to your apps will reduce overlay connection latency.
Edge routers do communicate with controllers when establishing connections, so you want HA controllers close to your routers.