Hi everyone,
I'm designing a 3-controller HA deployment and want to make sure I fully understand the failure modes before going to production.
The failure scenario I'm concerned about: 2 of 3 voting controllers go offline. The remaining controller C has lost quorum but is still reachable by edge clients.
From the documentation I understand that:
- C can still serve reads from its local data model (possibly stale)
- C cannot forward writes to the leader since no leader can be elected
- Existing circuits will remain up but can't be rerouted
What I haven't been able to determine from the docs is how session management behaves on the remaining controller:
- Can new clients authenticate and create API sessions against C?
- Can already-authenticated clients create new service sessions (e.g., to access a service they haven't dialed yet)?
- Does new circuit creation require a Raft commit, or can C handle it locally?
- If sessions do require writes, do clients get a clear error, or do they hang/timeout?
Basically: if C is the only controller my clients can see, what exactly can they still do and what breaks?
Thanks in advance for any insight!