I’m looking to set up a network across 3 cloud regions and have some questions about how the different components should be able to see each other on the underlying network.
For HA controller > controller communication, must every controller be able to see every other, or is this only for those which are voting in the consensus algorithm? (For example, could a non-voting connector have access only to one other?)
For Router > Controller, my question is similar - do routers need to be able to see all controllers, just voting controllers, or will they be fine with only seeing e.g. the controllers in the same cloud region as them.
And finally for Router > Router, I know they don’t need total visibility, but do they all have to be in one “mesh network” overall? Again I’m wondering if it’s possible for routers in one cloud region to have underlay connections only to those in the same region, in essence having 3 separate router meshes. Since the majority of traffic would stay within the same region this may be a possible path forwards.
Every voting member needs to be able to reach the others in at least one direction. I haven't done a lot of testing with various permutations, but at least in theory as long as you can dial in one direction, it should be able to establish a controller mesh.
I think the same is true of non-voting members. Since any voting member can become leader, the non-voting members needs to be able to connect to all voting members, but the connection can be establish in either direction.
If you test this and find it doesn't work, let me know. Right now we're relying on the Raft library to initiate the connections. We could always attempt to establish connections per-emptively if necessary.
Since controllers currently manage setting up routes, a controller needs to be connected to all routers that it wants to send routing updates to. So not every router needs to talk to every controller, but if it does limit the routing options. We also currently don't have the notion of controller groups, so all routers will attempt to connect to all routers, which may result in log spam.
We do have link groups, so you can control which routers a given router attempts to dial.