Came across a scenario in our test network where a service was flowing across an edge router (fabric/transit) that was in a different VPC than we wanted it to. This seems to be due to the fact that they are both in the same region so the latency is within 0.5ms, so the smart routing was picking the alternate path. Normally this wouldn’t be an issue, but with the traffic flows in this way it could incur additional cost.
Is there a way to allow services to use a particular path in the mesh, besides manually changing the link cost on certain paths? Also it seems like the docs don’t actually cover this, I found an old thread that was using different older commands to do this?
For reference of what we did to solve the issue for now:
Orig link cost of 5
ziti fabric list links
│ 1APP8hBEry935Rx5aftxUK │ er-usw2-fsx1 │ er-usw2-2 │ 1 │ 2.9ms │ 2.9ms │ Connected │ up │ 5 │
Manually adjusting link cost:
ziti fabric update link "1APP8hBEry935Rx5aftxUK" --static-cost 45
New link cost total of 49:
ziti fabric list links
│ 1APP8hBEry935Rx5aftxUK │ er-usw2-fsx1 │ er-usw2-2 │ 45 │ 2.8ms │ 2.9ms │ Connected │ up │ 49 │
Are you trying to keep all services off that router unless necessary? And/Or do you want that router to not be used in transit at all, only initiating and terminating services?
To move traffic away unless there is some other failure, you can cost up the router itself, which would bias paths away from it. This will affect all paths coming and going on all links that router has. Alternatively, you could set the no-transit property, and the controller won’t route through it, but it will still be allowed to dial and bind services.
To cost the router, use:
ziti fabric update router --cost
To mark the router no traversal, use :
ziti fabric update router --no-traversal
1 Like
Hello @pixitha ,
I see @mike.gorman responded while I was writing this, and he pointed out the two features that might be helpful, already 
I’ll just add a bit of extra context.
First, a caution, that link costs are ephemeral. In other words, if a link dies and is recreated, it’s a new link and doesn’t retain any data from the old link. This makes link costing only useful if you’ve got an external process monitoring link events and setting costs as they are created. We’ve talked about adding some sort of link rules concept, mostly to solve this costing issue.
We don’t have any per-service solutions. We’ve discussed moving service router policy into the fabric layer and allow it to affect not just the initiator and terminator routers but also transit routers. We may still end up doing this, but we’ll have to look at in the context of the routing refactoring we’ll be doing post-HA.
Let us know if link rules would be sufficient, or if you’d really need per-service routing rules. Or maybe per-service link costs? After all sometimes you want to allow links to be used, but only when there are no other options. We can use your needs to help figure out features and prioritization post-HA.
Cheers,
Paul
1 Like
In the way we would envision the usage, using a single “network” within ziti, but having distinct services that would only use certain routers. Think of each pod/team having their own gear, but its all centrally managed. This becomes a problem with AWS due to the latency being near identical across multiple different VPCs.
In this design, we would expect a service to be mapped to an individual teams transit routers/edge routers so they can keep their billing within their own AWS account.
Does that make sense?
Yes, makes sense. My currently thinking is that we’d want some entity to track this, like a router group, and then be able to associate a service with a single router group. Some mechanism which maybe had limited flexibility, but made it possible to maintain the mappings in the router. I’ll have to think about it some more.
Thank you for the insight into your use case.
I wanted to bump this thread and see if there has been any planning around this yet?
Paul’s been working on the link refactoring in this PR but doesn’t expect to merge it until early August.