How to bind each entry of host.v2 to an edge router?

nickchen120235 · October 22, 2024, 1:31pm

I am trying a multiple tunneler-enabled edge router setup in my ziti network as the following diagram.

I have one service "web" which contains both "web servers". I've tried the following host.v1 config:

{
  "protocol": "tcp",
  "address": "127.0.0.1",
  "port": 8080,
  "httpChecks": [],
  "portChecks": []
}

and the traffic is routed randomly when I set the terminator strategy to "random".

Now I want to use "weighted" and for example distribute the traffic evenly. I've tried the following host.v2 config:

"terminators": [
  {
    "address": "127.0.0.1",
    "listenOptions": {
      "cost": 1,
      "identity": "japan-gcore"
    },
    "port": 8080,
    "protocol": "tcp"
  },
  {
    "address": "127.0.0.1",
    "listenOptions": {
      "cost": 1,
      "identity": "gcp"
    },
    "port": 8080,
    "protocol": "tcp"
  }
]

and associate it with the service.

However I cannot access the servers and the router log on japan-gcore suggests that there is no terminator for the service

Oct 22 11:40:51 nick-vps ziti[1455168]: {"_context":"ch{edge}-\u003eu{classic}-\u003ei{MJAK}","chSeq":858,"connId":109,"edgeSeq":0,"error":"service 74DQjoretv4lcRI048NXbM has no terminators for instanceId ","file":"github.com/openziti/ziti/router/xgress_edge/listener.go:199","func":"github.com/openziti/ziti/router/xgress_edge.(*edgeClientConn).processConnect","level":"warning","msg":"failed to dial fabric","time":"2024-10-22T11:40:51.854Z","token":"c76c3177-7034-448b-bd18-2596a606b1b9","type":"EdgeConnectType"}

How do I set this up?

Also how does identity and bindUsingEdgeIdentity work in Listen Options host.v1?

plorenz · October 22, 2024, 3:15pm

Your initial setup is mostly likely correct. The simple host.v1 config you have will create a terminator on each edge router, each going to the local web server. To change the load balancing you only need to change the terminator strategy. I would recommend using the smartrouting strategy, as that will generally do a good job of balancing across the servers. Under light load, it will often bias to one side. If you only have a single request at a time, they might all go to the same web server. Once you have concurrent load, it should be balanced. weighted should also work reasonably well for the use case.

Note that both smart routing and weighted will use the full path cost in their calculations.

If you assign an identity to a terminator, that sets the instanceId on the terminator. If you use bindUsingEdgeIdentity, the value that gets assigned to the instanceId will be the name of the identity creating the terminator.

When a terminator has an instanceId set, the instanceId must be specified when dialing the service. This is used when you have a bunch of distinct resources that you've made available for the same service. Examples would be a VoIP service or an ssh service covering multiple hosts.

In the ssh example, you might use the hostname as the terminator identity. Then you could dial <hostname>@ssh to get to a specific host using the ssh service.

Hope that's helpful,
Paul

nickchen120235 · October 22, 2024, 4:32pm

Hi thanks for the reply.

How about the two web servers are hosted under different addresses? Like for example I change the "web server" under "gcore" to 127.0.0.1:9090. How do I setup the host.v1 (or maybe host.v2) config? I think in real world application we may have a setup like 10.0.0.100:8080 at site A and 192.168.0.100:8081 at site B and they have to be under the same service.

Is it the hosting cost under ZAC/Identity (or defaultHostingCost under API)?

Can you elaborate more on this? So if I have a host.v2 like above, how do I, for example, access the web server under the japan-gcore identity if my intercept.v1 is set to tcp:web.example.zt:80? My "gcore" edge router has an identity of japan-gcore, while my "gcp" edge router has an identity of gcp.

plorenz · October 22, 2024, 6:24pm

How about the two web servers are hosted under different addresses? Like for example I change the "web server" under "gcore" to 127.0.0.1:9090. How do I setup the host.v1 (or maybe host.v2) config? I think in real world application we may have a setup like 10.0.0.100:8080 at site A and 192.168.0.100:8081 at site B and they have to be under the same service.

You can specify a different host config for each router, if necessary. I think most people co-locate tunnelers with the services, so they don't need to do that, but it's an option if you don't wish to.

$ ziti edge update identity-configs --help
for the specified identity, use the given config for the given service

Usage:
 ziti edge update identity-configs <identity id or name> <service id or name> <config id or name> [flags]

You would create two configs, one for each web server, and assign them to the appropriate edge router/tunneler.

Is it the hosting cost under ZAC/Identity (or defaultHostingCost under API)?

Routing cost is calculated from the following components:

Link cost - calculated from the latency in ms
Router cost - assignable via the API
Static terminator cost - assignable via the API. Also set by the tunneler if defaultHostingCost is set or a cost is set in ListenOptions.
Dynamic terminator cost - The terminator strategies will adjust cost dynamically. Each active circuit increases cost by 2. Each failure will increase cost by 20 (this cost will gradually reduce).

Can you elaborate more on this? So if I have a host.v2 like above, how do I, for example, access the web server under the japan-gcore identity if my intercept.v1 is set to tcp:web.example.zt:80? My "gcore" edge router has an identity of japan-gcore, while my "gcp" edge router has an identity of gcp.

Take a look at the intercept.v1 dial options.

"dialOptions": {
            "additionalProperties": false,
            "properties": {
                "connectTimeoutSeconds": {
                    "$ref": "#/definitions/timeoutSeconds",
                    "description": "defaults to 5 seconds if no dialOptions are defined. defaults to 15 if dialOptions are defined but connectTimeoutSeconds is not specified."
                },
                "identity": {
                    "description": "Dial a terminator with the specified identity. '$dst_protocol', '$dst_ip', '$dst_port are resolved to the corresponding value of the destination address.",
                    "type": "string"
                }
            },
            "type": "object"
        },

They are unfortunately somewhat limited, since there's limited context available in the tunneler. So you can set the identity to some combination of the intercepted ip/port/protocol.

Alternately, you can create one intercept ip or hostname per target host and set the identity directly.

Paul

nickchen120235 · October 23, 2024, 4:28am

plorenz:

You can specify a different host config for each router, if necessary. I think most people co-locate tunnelers with the services, so they don't need to do that, but it's an option if you don't wish to.
$ ziti edge update identity-configs --help
for the specified identity, use the given config for the given service

Usage:
 ziti edge update identity-configs <identity id or name> <service id or name> <config id or name> [flags]
You would create two configs, one for each web server, and assign them to the appropriate edge router/tunneler.

Is this setting accessible from ZAC or a command-line-only thing?

So as long as the two identity between host.v2 (or host.v1) and intercept.v1 match I can use the specified intercept target (be it a domain or IP) to access the specific server under the same service?

TheLumberjack · October 23, 2024, 11:14am

Jumping in here real quick...

I'm pretty sure this is not something exposed in the ZAC at this time. I've actually never seen it myself until just now. If it's in the ZAC, I don't realize it. @rgalletto - there's no ZAC page for "identity-configs" yet, is there?

I don't think you need to worry about addressable terminators just yet. Addressable terminators allow you to basically connect to a specific identity. Your question here, assuming I follow along, is more "how do i make sure the right identity binds the service". That you control through service policies. You simply inform the overlay using service bind policies that an identity should host/bind a specific service and it'll take care of the rest for you. I think you should/could do this with just a regular bind service policy.

Addressable terminators allow you to effectively specify the "name" of a specific termination point. Think about ssh. For ssh, you don't want to have to create thousands of services, each with a bespoke intercept address. Instead, you want to be able to target one specific destination or terminator. With OpenZiti, the identity can inform the overlay what that address or unique name is. So when my local tunneler spins up, it can tell the overlay, "bind the ssh service, but add this extra bit of information, the instanceId, to the terminator and call it clint-ssh". Then when yours comes online, you too can bind the ssh service, but call it "nick-ssh". At this point, there are two terminators in the overlay: clint-ssh, nick-ssh. This allows a client to choose exactly which identity (or terminator) to target. Dial "clint-ssh" or "nick-ssh".

Now when you use bindUsingEdgeIdentity with tunnelers, that's just a shortcut to the configuration saying "whatever my name is, use that name as the terminator instanceId". You could also set it yourself if you prefer. This does mean that I could advertise "nick-ssh" from my identity. That's a different problem we'll be looking to correct sometime in the future.

Hopefully that extra context helps?

nickchen120235 · October 23, 2024, 1:47pm

Oh so the "identity" in configs (host.v1, intercept.v1, etc.) is not the "identity" we create and dial/bind the service, am I right? I was confused between the two and thought it would solve my problem. The ssh example is very clear, though I still don't understand the relationship between bindUsingEdgeIdentity, the "identity" in configs, and the "identity" we dial/bind the service with.

In the meantime I'll try identity-configs and see what will happen. I'll also take note that maybe co-locating the servers together is easier.

TheLumberjack · October 23, 2024, 2:06pm

Yeah. it should be instanceId, but with backward compatibility in mind, it's still "identity". It's the "instanceId of the terminator you want to send data to".... With that in mind...

As for bindUsingEdgeIdentity, think of it this way. This field will put the name of the identity into the instanceId of the terminator automatically. You change the identity name, the instanceId changes when the identity rebinds.

Take this example of an identity that binds with bindUsingEdgeIdentity. Here's my terminator:

ziti edge list terminators
╭───────────────────────┬─────────┬──────────────────────────────┬─────────┬───────────────────────┬───────────────┬──────┬────────────┬──────────────╮
│ ID                    │ SERVICE │ ROUTER                       │ BINDING │ ADDRESS               │ IDENTITY      │ COST │ PRECEDENCE │ DYNAMIC COST │
├───────────────────────┼─────────┼──────────────────────────────┼─────────┼───────────────────────┼───────────────┼──────┼────────────┼──────────────┤
│ i8jZtjX9La7GEPeX86GLe │ zsshSvc │ ip-172-31-47-200-edge-router │ edge    │ i8jZtjX9La7GEPeX86GLe │ zsshSvcServer │    0 │ default    │            0 │
╰───────────────────────┴─────────┴──────────────────────────────┴─────────┴───────────────────────┴───────────────┴──────┴────────────┴──────────────╯
results: 1-1 of 1

see that the IDENTITY field is zsshSvcServer? That's because this is the name of the identity and the host.v1 config indicated to bindUsingEdgeIdentity.

ziti edge show config zsshSvc.host.v1
{
    "address": "localhost",
    "listenOptions": {
        "bindUsingEdgeIdentity": true
    },
    "port": 22,
    "protocol": "tcp"
}

now i can make a new config and associate it to the zssh service:

ziti edge create config zsshSvc.host.v1.new host.v1 \
    '{"address": "localhost","listenOptions": {"identity":"some-random-value"},"port": 22,"protocol": "tcp"}'
ziti edge update service zsshSvc --configs 'zsshSvc.host.v1.new','zsshSvc.intercept.v1'

now when i look at terminators, you'll see a new terminator with "identity" of "some-random-value".

ziti edge list terminators
╭───────────────────────┬─────────┬──────────────────────────────┬─────────┬───────────────────────┬───────────────────┬──────┬────────────┬──────────────╮
│ ID                    │ SERVICE │ ROUTER                       │ BINDING │ ADDRESS               │ IDENTITY          │ COST │ PRECEDENCE │ DYNAMIC COST │
├───────────────────────┼─────────┼──────────────────────────────┼─────────┼───────────────────────┼───────────────────┼──────┼────────────┼──────────────┤
│ i8jZtjX9La7GEPeX86GLe │ zsshSvc │ ip-172-31-47-200-edge-router │ edge    │ i8jZtjX9La7GEPeX86GLe │ zsshSvcServer     │    0 │ default    │            0 │
│ tYx0AZ1F7jOgXkS1MDjA  │ zsshSvc │ ip-172-31-47-200-edge-router │ edge    │ tYx0AZ1F7jOgXkS1MDjA  │ some-random-value │    0 │ default    │            0 │
╰───────────────────────┴─────────┴──────────────────────────────┴─────────┴───────────────────────┴───────────────────┴──────┴────────────┴──────────────╯
results: 1-2 of 2

That help?

nickchen120235 · October 23, 2024, 2:42pm

Ah so with bindUsingEdgeIdentity we assign the "identity" (or actually the name of it) used to bind the service to be the instanceId (or "identity" under the config scope) of the terminator, which we then can use it in intercept.v1. That solves my confusion. Thanks for the explanation!

nickchen120235 · October 24, 2024, 7:20am

OK I think it is working now. Here's my setup.

Create two host.v1 configs for each server.

web-gcore.hostv1.conf

{
  "protocol": "tcp",
  "address": "127.0.0.1",
  "port": 9090,
  "httpChecks": [],
  "portChecks": []
}

web-gcp.hostv1.conf

{
  "protocol": "tcp",
  "address": "127.0.0.1",
  "port": 8080,
  "httpChecks": [],
  "portChecks": []
}

Assign them to each identity and the service.

ziti edge update identity-configs japan-gcore web web-gcore.hostv1.conf
ziti edge update identity-configs gcp web web-gcp.hostv1.conf

I keep the original intercept.v1.

Although in ZAC it only shows I have associated the intercept.v1, I do have two terminators for the service.

Thanks Paul and Clint for the help!

nickchen120235 · October 24, 2024, 7:41am

By the way how do I delete these configs? I've tried

ziti edge delete config where 'name="web-gcp.hostv1.conf"'

but it failed with the following message:

nickchen120235@nick-gcp:~$ ziti edge delete config where 'name="web-gcp.hostv1.conf"'
filter returned results: 1-1 of 1
delete of config with id p45YcGdb8HtNTmCl3C2nt: FAIL
error: error deleting configs/p45YcGdb8HtNTmCl3C2nt instance in Ziti Edge Controller at https://localhost:1280/edge/management/v1. Status code: 500 Internal Server Error, Server returned: {
    "error": {
        "cause": {
            "code": "UNHANDLED",
            "message": "config is in use by identity 3xNSEpPON for service 74DQjoretv4lcRI048NXbM"
        },
        "code": "UNHANDLED",
        "message": "An unhandled error occurred",
        "requestId": "DoyzE4vO2"
    },
    "meta": {
        "apiEnrollmentVersion": "0.0.1",
        "apiVersion": "0.0.1"
    }
}

TheLumberjack · October 24, 2024, 11:41am

Status code: 500 Internal Server Error, Server returned:

INTERESTING! If you can provide steps to reproduce, that's clearly a bug. Anytime you see a 500 error, it's a problem we need to fix.

Try deleting it by ID instead and see what happens?

Is the config associated to any service? Maybe it's just a bad looking for a dependency of that nature?

nickchen120235 · October 24, 2024, 12:31pm

No luck unfortunately. The controller returns the same error.

Sure! Here's what I've done so far.

Create a service
Associate host.v1 config using ziti edge update identity-configs <identity> <config> <service> where <service> is the one created in step 1.
Delete the service created in step 1.
Try to delete the config created in step 2 using ziti edge delete config where 'name=<config>'.

Also the configs associated that way does not show up in ziti edge list service configs <id or name> (or ziti edge list config services <id or name> respectively).

TheLumberjack · October 24, 2024, 12:33pm

Ahhh - identity-configs is in the mix. @plorenz maybe you can have a peek?

plorenz · October 24, 2024, 1:39pm

I should be able to duplicate this. I'll give it a try and see what I find. I suspect it's a constraints issue in the DB.

Paul

plorenz · October 28, 2024, 7:05pm

Found the issue. Fix should hopefully be in the next release.

Cheers,
Paul

Topic		Replies	Views
Adding port to Intercept or Host adds another terminator General Questions	5	262	November 7, 2022
How do I create a service that connects to a Subnet via a router instead of just a single host?	83	1994	November 14, 2022
Best Way for Providing HA Access Ziti Overlay	13	466	May 20, 2023
Using edge router with the tunneler enabled Ziti Overlay	1	341	June 23, 2022
Request for examples for service configurations General Questions	4	421	November 16, 2022

How to bind each entry of host.v2 to an edge router?

Related topics