So, assuming I have multiple services and I want to provide them some kind of High Availability. One way to archive this would be creating 2 ziti-routers with the same bind policy. And both of them in the same Dial policy for the clients.
Am I right? Is there a better way to archive this without SDK use, considering we’re not holding the source code of all of them?
There are multiple kinds of HA to cover. I’m going to split this response over a series of of posts, one for each scenario. I’m going to cover this from the perspective of using routers with embedded tunnelers. This can also be done with standalone tunnelers, with some slight variation in configuration.
First scenario: Making sure the router isn’t a single point of failure.
Let’s start with making sure we can access the server from multiple edge routers. To do this, we allow multiple tunneler/routers to host the service.
ziti edge create config test-host-config host.v2 '
{
"terminators" : [
{ "address": "192.168.3.136", "port" : 22, "protocol": "tcp" }
]
}
'
ziti edge create service test -c test-host-config --terminator-strategy smartrouting
ziti edge create edge-router edge-router-1 --tunneler-enabled
ziti edge create edge-router edge-router-2 --tunneler-enabled
# skipping the router enrollment step
ziti edge update identity edge-router-1 --role-attributes 'test-hosts'
ziti edge update identity edge-router-2 --role-attributes 'test-hosts'
ziti edge create service-edge-router-policy test-serp --service-roles '@test' --edge-router-roles '#all'
ziti edge create service-policy test-bind Bind --service-roles '@test' --identity-roles '#test-hosts'
If I then startup edge routers 1 and 2 and list terminators, I see:
$ ziti fabric list terminators
╭──────┬─────────┬───────────────┬─────────┬──────────────────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├──────┼─────────┼───────────────┼─────────┼──────────────────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ d68X │ test │ edge-router-1 │ tunnel │ 87a5e273-2ae4-4ea9-b31a-4d4dc00f96e8 │ │ 0 │ default │ 0 │
│ zDAd │ test │ edge-router-2 │ tunnel │ f8b831cc-811a-4f3b-a795-1fbd46cfb425 │ │ 0 │ default │ 0 │
╰──────┴─────────┴───────────────┴─────────┴──────────────────────────────────────┴──────────┴──────┴────────────┴──────────────╯
Our two edge routers are both able to provide access to our single server.
A few things to note:
default set. This means we’ll use them both. When creating the service we selected the smart routing terminator strategy. This will pick the lowest cost path for each new connection established. We could also have used weighted or random (see Ziti Services | Ziti) for more details.Next up, we’ll cover HA with multiple servers in a non-failover setup.
The next scenario to look at is when we have multiple servers. Let’s say in this case we’re accessing a set of web service providers, so it doesn’t matter which one we hit. Let’s assume we always want access to the service via at least two routers because we don’t want any single points of failure in our solution.
The simplest way to do this is to just add another address to the config:
ziti edge create config test-host-config host.v2 '
{
"terminators" : [
{ "address": "192.168.3.136", "port" : 8080, "protocol": "tcp" },
{ "address": "192.168.3.137", "port" : 8080, "protocol": "tcp" }
]
}
'
ziti edge create service test -c test-host-config --terminator-strategy smartrouting
ziti edge create edge-router edge-router-1 --tunneler-enabled
ziti edge create edge-router edge-router-2 --tunneler-enabled
# skipping router enrollment steps
ziti edge update identity edge-router-1 --role-attributes 'test-hosts'
ziti edge update identity edge-router-2 --role-attributes 'test-hosts'
ziti edge create service-edge-router-policy test-serp --service-roles '@test' --edge-router-roles '#all'
ziti edge create service-policy test-bind Bind --service-roles '@test' --identity-roles '#test-hosts'
Now when we list terminators, we’ll see four. Each router will have a terminator for each server.
$ ziti fabric list terminators
╭──────┬─────────┬───────────────┬─────────┬──────────────────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├──────┼─────────┼───────────────┼─────────┼──────────────────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ ZAOz │ test │ edge-router-1 │ tunnel │ f20bde9b-7cd6-4a8a-a1f4-8f4cd1079277 │ │ 0 │ default │ 0 │
│ dnmz │ test │ edge-router-2 │ tunnel │ 566e2c00-fd4b-45a7-86bf-a53515eb09ce │ │ 0 │ default │ 0 │
│ doRz │ test │ edge-router-1 │ tunnel │ bc6f7c56-ad7a-4610-b771-805f82f6fb70 │ │ 0 │ default │ 0 │
│ dyVZ │ test │ edge-router-2 │ tunnel │ 53cedebd-d9da-44d9-8458-3f16fb0811a9 │ │ 0 │ default │ 0 │
╰──────┴─────────┴───────────────┴─────────┴──────────────────────────────────────┴──────────┴──────┴────────────┴──────────────╯
results: 1-4 of 4
If one of the servers goes down, connections to it will fail. This will increase the cost of that terminator and the other server will be preferred. The cost will slowly come down and eventually it will be retried. If it’s back up, we’ll start using it again. If it’s still down, the cost will jump back up.
We can be more proactive with our health checking though, by defining some health checks in the server config.
ziti edge update config test-host-config host.v2 --data '
{
"terminators" : [
{
"address": "192.168.3.136",
"port" : 8080,
"protocol": "tcp",
"portChecks" : [
{
"address" : "192.168.3.136:8080",
"interval" : "5s",
"timeout" : "100ms",
"actions" : [
{
"trigger" : "fail",
"consecutiveEvents" : 3,
"action" : "mark unhealthy"
},
{
"trigger" : "pass",
"consecutiveEvents" : 3,
"action" : "mark healthy"
}
]
}
]
},
{
"address": "192.168.3.137",
"port" : 8080,
"protocol": "tcp",
"portChecks" : [
{
"address" : "192.168.3.137:8080",
"interval" : "5s",
"timeout" : "100ms",
"actions" : [
{
"trigger" : "fail",
"consecutiveEvents" : 3,
"action" : "mark unhealthy"
},
{
"trigger" : "pass",
"consecutiveEvents" : 3,
"action" : "mark healthy"
}
]
}
]
}
]
}
'
This defines a simple port check which will run every 5 seconds. If the connect fails or times out, the check will fail. After 3 consecutive checks fail, the terminator will be marked as failed, which will it prevent it from being used, as long as there are any other healthy terminators. The health check will continue to run and once it has three checks pass in a row, the terminator will be returned to its original precedence of default.
In addition to simple port checks, ziti also supports http checks.
Next we’ll take a look at how to handle failover when you’ve got a primary server that should always be used until it fails, at which point you want to fail over to a secondary.
The final scenario is HA with failover, where you’ve got a primary and a secondary. For this example we’re going to use the standalone tunneler, rather than the router embedded tunneler, for a reason I’ll explain at the end.
This will look a little different. Here is the script to create the setup:
ziti edge create config test-host-config host.v2 '
{
"terminators" : [
{
"address": "localhost",
"port" : 8080,
"protocol": "tcp",
"portChecks" : [
{
"address" : "localhost:8080",
"interval" : "5s",
"timeout" : "100ms",
"actions" : [
{
"trigger" : "fail",
"consecutiveEvents" : 3,
"action" : "mark unhealthy"
},
{
"trigger" : "pass",
"consecutiveEvents" : 3,
"action" : "mark healthy"
}
]
}
]
}
]
}
'
ziti edge create service test -c test-host-config --terminator-strategy smartrouting
ziti edge create identity service test-tunnel-1 --role-attributes 'test-hosts' --service-precedences test=required -o host1.jwt
ziti edge create identity service test-tunnel-2 --role-attributes 'test-hosts' --service-precedences test=default -o host2.jwt
ziti edge enroll -j host1.jwt -o host1.json
ziti edge enroll -j host2.jwt -o host2.json
ziti edge create edge-router-policy test-routers --edge-router-roles '#all' --identity-roles '#test-hosts'
ziti edge create service-edge-router-policy test-serp --service-roles '@test' --edge-router-roles '#all'
ziti edge create service-policy test-bind Bind --service-roles '@test' --identity-roles '#test-hosts'
Notes:
test service on each hosting identity.$ ziti fabric list terminators
╭──────┬─────────┬───────────────┬─────────┬─────────────────────────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├──────┼─────────┼───────────────┼─────────┼─────────────────────────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ Z0xZ │ test │ edge-router-2 │ edge │ hosted:a9bf2ad7-d784-40ba-847b-6cb04ecee1ca │ │ 0 │ required │ 0 │
│ ZBwX │ test │ edge-router-1 │ edge │ hosted:6796071b-c866-46da-b75b-78afdcc094ff │ │ 0 │ default │ 0 │
│ ZOvd │ test │ edge-router-1 │ edge │ hosted:a9bf2ad7-d784-40ba-847b-6cb04ecee1ca │ │ 0 │ required │ 0 │
│ zg2X │ test │ edge-router-2 │ edge │ hosted:6796071b-c866-46da-b75b-78afdcc094ff │ │ 0 │ default │ 0 │
╰──────┴─────────┴───────────────┴─────────┴─────────────────────────────────────────────┴──────────┴──────┴────────────┴──────────────╯
results: 1-4 of 4
So, the reason I showed this setup with the standalone tunneler and not the router-embedded tunneler is because in doing these example I found that we have a gap. You can’t currently set precedence or cost in the config, which means for multiple terminators configured on the same router, you can’t differentiate them. I’ve put in an issue to track this, and it should hopefully be resolved shortly: Add precedence/cost back to host.v2 · Issue #931 · openziti/edge · GitHub
Let me know if this is helpful, or where further clarification could be helpful.
Finally, just a note that doing this with an SDK enabled server application is simpler from a configuration perspective. No service config is required. If desired, the SDK application can have cost and precedence configured on its identity or it can configure its own cost and precedence. The application will also bind on multiple routers (if available), so you get redundancy there as well.
just wondering @plorenz if that’s all you have to offer??? 
Might you have more to say on this matter?
This is awesome!
Thanks so much Paul.
This really shows the capabilities to support not only elastic architectures, but it show us the ability of OpenZiti to distribute our nodes and allow us full H-A to support and access them.
Really impressed!!
Thanks so much once again!
Thank you for asking, @TheLumberjack, as it happens, I did have a follow up.
With the fix for the issue above in place, we can take a look at how to handle the failover case in the router-embedded tunneler environment. It looks almost identical to the second case above, but we add a couple of lines to the config:
ziti edge create config test-host-config host.v2 '
{
"terminators" : [
{
"address": "192.168.3.136",
"port" : 8080,
"protocol": "tcp",
"portChecks" : [
{
"address" : "192.168.3.136:8080",
"interval" : "5s",
"timeout" : "100ms",
"actions" : [
{
"trigger" : "fail",
"consecutiveEvents" : 3,
"action" : "mark unhealthy"
},
{
"trigger" : "pass",
"consecutiveEvents" : 3,
"action" : "mark healthy"
}
]
}
],
"listenOptions" : {
"precedence" : "required"
}
},
{
"address": "192.168.3.137",
"port" : 8080,
"protocol": "tcp",
"portChecks" : [
{
"address" : "192.168.3.137:8080",
"interval" : "5s",
"timeout" : "100ms",
"actions" : [
{
"trigger" : "fail",
"consecutiveEvents" : 3,
"action" : "mark unhealthy"
},
{
"trigger" : "pass",
"consecutiveEvents" : 3,
"action" : "mark healthy"
}
]
}
]
}
]
}
'
Specifically, we add these couple of line to just the first terminator, which is our primary.
"listenOptions" : {
"precedence" : "required"
}
When we run with this setup and look at our terminators, you’d see something like:
$ ziti fabric list terminators
╭──────┬─────────┬───────────────┬─────────┬──────────────────────────────────────┬──────────┬──────┬────────────┬──────────────╮
│ ID │ SERVICE │ ROUTER │ BINDING │ ADDRESS │ IDENTITY │ COST │ PRECEDENCE │ DYNAMIC COST │
├──────┼─────────┼───────────────┼─────────┼──────────────────────────────────────┼──────────┼──────┼────────────┼──────────────┤
│ 8Vj8 │ test │ edge-router-1 │ tunnel │ db774930-96bb-4334-b03b-814897419084 │ │ 0 │ required │ 0 │
│ ErmK │ test │ edge-router-2 │ tunnel │ 40553c88-f30c-457e-be5f-72730d3fd73e │ │ 0 │ required │ 0 │
│ KjNK │ test │ edge-router-2 │ tunnel │ 8cc16550-02da-4e48-aa79-ccad0fe7448e │ │ 0 │ default │ 0 │
│ KxDK │ test │ edge-router-1 │ tunnel │ 4c066d0e-fb8d-4538-9601-860b5d02db23 │ │ 0 │ default │ 0 │
╰──────┴─────────┴───────────────┴─────────┴──────────────────────────────────────┴──────────┴──────┴────────────┴──────────────╯
results: 1-4 of 4
So you can see each router has one required terminator, going to the primary, and one default, going to the secondary. If the health check on the primary fails, the precedence will be dropped to failed and traffic will be directed to the secondary. When the health check on the primary recovers, it will return to precedence required and traffic will go back to those terminators.
If for some reason, traffic between one of the routers and the primary was disrupted, only one of the two terminators would get dropped to failed. Since we would still have a terminator at required going to the primary, that terminator would be used on its own, until connectivity was restored from the other router.
This fix for the issue will be out in the next Ziti release, v0.25.3, likely in the next day or two.
Cheers @natashell ! Thanks for asking since we can hopefully to use these posts as a basis for improving our core documentation, which is lacking a bit in this area 
This describes well how to setup multiple connections to multiple services and having multiple routers. What about multiple controllers?
Is there a description somewhere about how to setup HA for the controller and PKI?
This is literally something being worked on right now to be released very soon. @plorenz may have better insight for you.
Other than “it’s coming soon”, I don’t think there’s anything at all to add… I’m sure there’ll be a post/tweet/ziti tv about it when it launches …
If you want to get some insight into the work being done, there’s a video covering it here: Ziti Kitchen - Mar 30 2022 - YouTube
Any progress on multiple controllers support?