Best Way for Providing HA Access

So, assuming I have multiple services and I want to provide them some kind of High Availability. One way to archive this would be creating 2 ziti-routers with the same bind policy. And both of them in the same Dial policy for the clients.

Am I right? Is there a better way to archive this without SDK use, considering weโ€™re not holding the source code of all of them?

There are multiple kinds of HA to cover. Iโ€™m going to split this response over a series of of posts, one for each scenario. Iโ€™m going to cover this from the perspective of using routers with embedded tunnelers. This can also be done with standalone tunnelers, with some slight variation in configuration.

First scenario: Making sure the router isnโ€™t a single point of failure.

Letโ€™s start with making sure we can access the server from multiple edge routers. To do this, we allow multiple tunneler/routers to host the service.

ziti edge create config test-host-config host.v2 '
{
        "terminators" : [
                { "address": "192.168.3.136", "port" : 22, "protocol": "tcp" }
        ]
}
'

ziti edge create service test -c test-host-config  --terminator-strategy smartrouting

ziti edge create edge-router edge-router-1 --tunneler-enabled 
ziti edge create edge-router edge-router-2 --tunneler-enabled

# skipping the router enrollment step

ziti edge update identity edge-router-1 --role-attributes 'test-hosts'
ziti edge update identity edge-router-2 --role-attributes 'test-hosts'

ziti edge create service-edge-router-policy test-serp --service-roles '@test' --edge-router-roles '#all'

ziti edge create service-policy test-bind Bind --service-roles '@test' --identity-roles '#test-hosts' 

If I then startup edge routers 1 and 2 and list terminators, I see:

$ ziti fabric list terminators
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ ID   โ”‚ SERVICE โ”‚ ROUTER        โ”‚ BINDING โ”‚ ADDRESS                              โ”‚ IDENTITY โ”‚ COST โ”‚ PRECEDENCE โ”‚ DYNAMIC COST โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ d68X โ”‚ test    โ”‚ edge-router-1 โ”‚ tunnel  โ”‚ 87a5e273-2ae4-4ea9-b31a-4d4dc00f96e8 โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ”‚ zDAd โ”‚ test    โ”‚ edge-router-2 โ”‚ tunnel  โ”‚ f8b831cc-811a-4f3b-a795-1fbd46cfb425 โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

Our two edge routers are both able to provide access to our single server.

A few things to note:

  1. When using router/tunneler combos, when the edge router has tunneler enabled, we create an identity with the same name as the edge router as a stand-in for the tunneler in service policies. This is why we set the attribute on the edge router identities rather than directly on the edge routers.
  2. Looking at the terminators, they have the same precedence level of default set. This means weโ€™ll use them both. When creating the service we selected the smart routing terminator strategy. This will pick the lowest cost path for each new connection established. We could also have used weighted or random (see Ziti Services | Ziti) for more details.

Next up, weโ€™ll cover HA with multiple servers in a non-failover setup.

1 Like

The next scenario to look at is when we have multiple servers. Letโ€™s say in this case weโ€™re accessing a set of web service providers, so it doesnโ€™t matter which one we hit. Letโ€™s assume we always want access to the service via at least two routers because we donโ€™t want any single points of failure in our solution.

The simplest way to do this is to just add another address to the config:

ziti edge create config test-host-config host.v2 '
{
        "terminators" : [
                { "address": "192.168.3.136", "port" : 8080, "protocol": "tcp" },
                { "address": "192.168.3.137", "port" : 8080, "protocol": "tcp" }
        ]
}
'

ziti edge create service test -c test-host-config --terminator-strategy smartrouting

ziti edge create edge-router edge-router-1 --tunneler-enabled 
ziti edge create edge-router edge-router-2 --tunneler-enabled

# skipping router enrollment steps

ziti edge update identity edge-router-1 --role-attributes 'test-hosts'
ziti edge update identity edge-router-2 --role-attributes 'test-hosts'
ziti edge create service-edge-router-policy test-serp --service-roles '@test' --edge-router-roles '#all'
ziti edge create service-policy test-bind Bind --service-roles '@test' --identity-roles '#test-hosts'

Now when we list terminators, weโ€™ll see four. Each router will have a terminator for each server.

$ ziti fabric list terminators
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ ID   โ”‚ SERVICE โ”‚ ROUTER        โ”‚ BINDING โ”‚ ADDRESS                              โ”‚ IDENTITY โ”‚ COST โ”‚ PRECEDENCE โ”‚ DYNAMIC COST โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ ZAOz โ”‚ test    โ”‚ edge-router-1 โ”‚ tunnel  โ”‚ f20bde9b-7cd6-4a8a-a1f4-8f4cd1079277 โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ”‚ dnmz โ”‚ test    โ”‚ edge-router-2 โ”‚ tunnel  โ”‚ 566e2c00-fd4b-45a7-86bf-a53515eb09ce โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ”‚ doRz โ”‚ test    โ”‚ edge-router-1 โ”‚ tunnel  โ”‚ bc6f7c56-ad7a-4610-b771-805f82f6fb70 โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ”‚ dyVZ โ”‚ test    โ”‚ edge-router-2 โ”‚ tunnel  โ”‚ 53cedebd-d9da-44d9-8458-3f16fb0811a9 โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
results: 1-4 of 4

If one of the servers goes down, connections to it will fail. This will increase the cost of that terminator and the other server will be preferred. The cost will slowly come down and eventually it will be retried. If itโ€™s back up, weโ€™ll start using it again. If itโ€™s still down, the cost will jump back up.

We can be more proactive with our health checking though, by defining some health checks in the server config.

ziti edge update config test-host-config host.v2 --data '
{
        "terminators" : [
                { 
                        "address": "192.168.3.136", 
                        "port" : 8080, 
                        "protocol": "tcp",
                        "portChecks" : [
                                {
                                        "address" : "192.168.3.136:8080",
                                        "interval" : "5s",
                                        "timeout" : "100ms",
                                        "actions" : [
                                                {
                                                        "trigger" : "fail",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark unhealthy"
                                                },
                                                {
                                                        "trigger" : "pass",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark healthy"
                                                }
                                        ]
                                }
                        ]
                },
                { 
                        "address": "192.168.3.137", 
                        "port" : 8080, 
                        "protocol": "tcp",
                        "portChecks" : [
                                {
                                        "address" : "192.168.3.137:8080",
                                        "interval" : "5s",
                                        "timeout" : "100ms",
                                        "actions" : [
                                                {
                                                        "trigger" : "fail",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark unhealthy"
                                                },
                                                {
                                                        "trigger" : "pass",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark healthy"
                                                }
                                        ]
                                }
                        ]
                }
        ]
}
'

This defines a simple port check which will run every 5 seconds. If the connect fails or times out, the check will fail. After 3 consecutive checks fail, the terminator will be marked as failed, which will it prevent it from being used, as long as there are any other healthy terminators. The health check will continue to run and once it has three checks pass in a row, the terminator will be returned to its original precedence of default.

In addition to simple port checks, ziti also supports http checks.

Next weโ€™ll take a look at how to handle failover when youโ€™ve got a primary server that should always be used until it fails, at which point you want to fail over to a secondary.

1 Like

The final scenario is HA with failover, where youโ€™ve got a primary and a secondary. For this example weโ€™re going to use the standalone tunneler, rather than the router embedded tunneler, for a reason Iโ€™ll explain at the end.

This will look a little different. Here is the script to create the setup:

ziti edge create config test-host-config host.v2 '
{
        "terminators" : [
                { 
                        "address": "localhost", 
                        "port" : 8080, 
                        "protocol": "tcp",
                        "portChecks" : [
                                {
                                        "address" : "localhost:8080",
                                        "interval" : "5s",
                                        "timeout" : "100ms",
                                        "actions" : [
                                                {
                                                        "trigger" : "fail",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark unhealthy"
                                                },
                                                {
                                                        "trigger" : "pass",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark healthy"
                                                }
                                        ]
                                }
                        ]
                }
        ]
}
'

ziti edge create service test -c test-host-config --terminator-strategy smartrouting

ziti edge create identity service test-tunnel-1 --role-attributes 'test-hosts' --service-precedences test=required -o host1.jwt
ziti edge create identity service test-tunnel-2 --role-attributes 'test-hosts' --service-precedences test=default -o host2.jwt

ziti edge enroll -j host1.jwt -o host1.json
ziti edge enroll -j host2.jwt -o host2.json

ziti edge create edge-router-policy test-routers --edge-router-roles '#all' --identity-roles '#test-hosts'
ziti edge create service-edge-router-policy test-serp --service-roles '@test' --edge-router-roles '#all'
ziti edge create service-policy test-bind Bind --service-roles '@test' --identity-roles '#test-hosts' 

Notes:

  1. The first thing you might notice is that the config only has one terminator in it. This is because we assume that we are running the tunneler on the same host as the service. This also lets us use localhost in the config. If we wanted to run the tunnelers on a different machine, we would have to create a config for each server and then override the default config for the test service on each hosting identity.
  2. As long as the tunneler hosting identities have access to multiple routers, we will still have router redundancy. I ran this with two routers and ended up with four terminators, two at default, two at required, one of each on each router. The output of that is below.
  3. In order to configure which tunneler was talking to the primary, we set the precedence on the identity. An identity can have a default precedence for services it is hostings and per-service precedence (and cost) can also be configured.
$ ziti fabric list terminators
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ ID   โ”‚ SERVICE โ”‚ ROUTER        โ”‚ BINDING โ”‚ ADDRESS                                     โ”‚ IDENTITY โ”‚ COST โ”‚ PRECEDENCE โ”‚ DYNAMIC COST โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ Z0xZ โ”‚ test    โ”‚ edge-router-2 โ”‚ edge    โ”‚ hosted:a9bf2ad7-d784-40ba-847b-6cb04ecee1ca โ”‚          โ”‚    0 โ”‚ required   โ”‚            0 โ”‚
โ”‚ ZBwX โ”‚ test    โ”‚ edge-router-1 โ”‚ edge    โ”‚ hosted:6796071b-c866-46da-b75b-78afdcc094ff โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ”‚ ZOvd โ”‚ test    โ”‚ edge-router-1 โ”‚ edge    โ”‚ hosted:a9bf2ad7-d784-40ba-847b-6cb04ecee1ca โ”‚          โ”‚    0 โ”‚ required   โ”‚            0 โ”‚
โ”‚ zg2X โ”‚ test    โ”‚ edge-router-2 โ”‚ edge    โ”‚ hosted:6796071b-c866-46da-b75b-78afdcc094ff โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
results: 1-4 of 4

So, the reason I showed this setup with the standalone tunneler and not the router-embedded tunneler is because in doing these example I found that we have a gap. You canโ€™t currently set precedence or cost in the config, which means for multiple terminators configured on the same router, you canโ€™t differentiate them. Iโ€™ve put in an issue to track this, and it should hopefully be resolved shortly: Add precedence/cost back to host.v2 ยท Issue #931 ยท openziti/edge ยท GitHub

Let me know if this is helpful, or where further clarification could be helpful.

Finally, just a note that doing this with an SDK enabled server application is simpler from a configuration perspective. No service config is required. If desired, the SDK application can have cost and precedence configured on its identity or it can configure its own cost and precedence. The application will also bind on multiple routers (if available), so you get redundancy there as well.

1 Like

just wondering @plorenz if thatโ€™s all you have to offer??? :stuck_out_tongue: Might you have more to say on this matter?

This is awesome!
Thanks so much Paul.

This really shows the capabilities to support not only elastic architectures, but it show us the ability of OpenZiti to distribute our nodes and allow us full H-A to support and access them.

Really impressed!!

Thanks so much once again!

Thank you for asking, @dovholuknf, as it happens, I did have a follow up.

With the fix for the issue above in place, we can take a look at how to handle the failover case in the router-embedded tunneler environment. It looks almost identical to the second case above, but we add a couple of lines to the config:

ziti edge create config test-host-config host.v2 '
{
        "terminators" : [
                { 
                        "address": "192.168.3.136", 
                        "port" : 8080, 
                        "protocol": "tcp",
                        "portChecks" : [
                                {
                                        "address" : "192.168.3.136:8080",
                                        "interval" : "5s",
                                        "timeout" : "100ms",
                                        "actions" : [
                                                {
                                                        "trigger" : "fail",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark unhealthy"
                                                },
                                                {
                                                        "trigger" : "pass",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark healthy"
                                                }
                                        ]
                                }
                        ],
                        "listenOptions" : {
                                "precedence" : "required"
                        }
                },
                { 
                        "address": "192.168.3.137", 
                        "port" : 8080, 
                        "protocol": "tcp",
                        "portChecks" : [
                                {
                                        "address" : "192.168.3.137:8080",
                                        "interval" : "5s",
                                        "timeout" : "100ms",
                                        "actions" : [
                                                {
                                                        "trigger" : "fail",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark unhealthy"
                                                },
                                                {
                                                        "trigger" : "pass",
                                                        "consecutiveEvents" : 3,
                                                        "action" : "mark healthy"
                                                }
                                        ]
                                }
                        ]
                }
        ]
}
'

Specifically, we add these couple of line to just the first terminator, which is our primary.

                        "listenOptions" : {
                                "precedence" : "required"
                        }

When we run with this setup and look at our terminators, youโ€™d see something like:

$ ziti fabric list terminators
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ ID   โ”‚ SERVICE โ”‚ ROUTER        โ”‚ BINDING โ”‚ ADDRESS                              โ”‚ IDENTITY โ”‚ COST โ”‚ PRECEDENCE โ”‚ DYNAMIC COST โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ 8Vj8 โ”‚ test    โ”‚ edge-router-1 โ”‚ tunnel  โ”‚ db774930-96bb-4334-b03b-814897419084 โ”‚          โ”‚    0 โ”‚ required   โ”‚            0 โ”‚
โ”‚ ErmK โ”‚ test    โ”‚ edge-router-2 โ”‚ tunnel  โ”‚ 40553c88-f30c-457e-be5f-72730d3fd73e โ”‚          โ”‚    0 โ”‚ required   โ”‚            0 โ”‚
โ”‚ KjNK โ”‚ test    โ”‚ edge-router-2 โ”‚ tunnel  โ”‚ 8cc16550-02da-4e48-aa79-ccad0fe7448e โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ”‚ KxDK โ”‚ test    โ”‚ edge-router-1 โ”‚ tunnel  โ”‚ 4c066d0e-fb8d-4538-9601-860b5d02db23 โ”‚          โ”‚    0 โ”‚ default    โ”‚            0 โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
results: 1-4 of 4

So you can see each router has one required terminator, going to the primary, and one default, going to the secondary. If the health check on the primary fails, the precedence will be dropped to failed and traffic will be directed to the secondary. When the health check on the primary recovers, it will return to precedence required and traffic will go back to those terminators.

If for some reason, traffic between one of the routers and the primary was disrupted, only one of the two terminators would get dropped to failed. Since we would still have a terminator at required going to the primary, that terminator would be used on its own, until connectivity was restored from the other router.

This fix for the issue will be out in the next Ziti release, v0.25.3, likely in the next day or two.

2 Likes

Cheers @natashell ! Thanks for asking since we can hopefully to use these posts as a basis for improving our core documentation, which is lacking a bit in this area :smile:

1 Like

This describes well how to setup multiple connections to multiple services and having multiple routers. What about multiple controllers?
Is there a description somewhere about how to setup HA for the controller and PKI?

This is literally something being worked on right now to be released very soon. @plorenz may have better insight for you.

Other than โ€œitโ€™s coming soonโ€, I donโ€™t think thereโ€™s anything at all to addโ€ฆ Iโ€™m sure thereโ€™ll be a post/tweet/ziti tv about it when it launches โ€ฆ

If you want to get some insight into the work being done, thereโ€™s a video covering it here: Ziti Kitchen - Mar 30 2022 - YouTube

2 Likes