ECMP logic across multiple fabric routers?

1

Is there any logic for ECMP across multiple fabric routers?

I tried to find a reference in the docs, but I may have searched for the wrong term.

thanks!

2

No, we’re not currently using ECMP. It’s something we’ve thought about, and we may experiment with it as we work on the next iteration of the fabric.

Cheers,
Paul

3

–terminator-strategy=“random”

This seems to allow some hacked amount of load balancing across multiple fabric routers, but its not very reliable FYI.

I noticed during some testing that traffic was being limited to ~650Mbit/sec of throughput, when I put the terminator to random (Away from smart) the traffic eventually picked up to 1.46Gbit/sec.

It seems that large fat flows would benefit from this greatly, especially since all of the single client connections are funneled through a single connection between routers.

4

The terminator strategy only applies to the terminator selection, which only affects the last part of the circuit. It doesn’t affect the rest of the path selection. The rest of the path selection is affected by the following:

  1. router cost
    1. Settable via the API
  2. link cost
    1. Settable via the API
    2. Needs to be reworked, as it’s tied to the link and links come and go, so isn’t properly persistent
  3. link latency
    1. Updated from the routers when routers report metrics in, configurable, but usually between 15s and 1 minute.

So when you’ve got a bunch of traffic, the main thing that will cause traffic to get shifted from one place to another is changes to latency, which will take a bit of time. As you saw, it will adapt but it can be a bit slow. This is something we’re looking at improving in the next version of fabric routing, by moving more routing intelligence from the controller to the routers.

Hope that’s helpful,
Paul

5

Should I put in an official feature request on github for this stuff? If so, which repo?

6

I started a GH Project for planning the next version of routing here: Routing.Next · GitHub and added an entry for multi-path.

Cheers,
Paul

7

Any update on the planning for ECMP? (or other types of load balancing)

8

Unfortunately, no. We’re very excited to get started on that, but it’s still gated behind HA.