Address multi-hosted service on specific host

Support General Questions
1

Openziti offers the possibility to bind a service to multiple hosts. When connecting to the service you will end up at one of the hosts available. This is fine for HA and load distribution.
But is it possible to configure the service in a way that, although it is bound to multiple hosts, to get connected to a specific host. Maybe by its identity + service address?

A solution is to create a service for each host. But if there are 1000 hosts that provide basically the same service, it would end up with 1000 services that need to be configured and maintained.

2

Hi @Chris, welcome to the community and to OpenZiti (and zrok and BrowZer).

Yes you can configure a service to a specific identity when using multiple identities. Did you find the relevant doc? Was it clear/helpful if you did? Ziti Services | OpenZiti

If you are interested in in a single service where 1000's of hosts can bind the same service, that's what we call 'addressable terminators'. A great example of this is zssh if you're looking for an example.

You can find zssh here: zssh/README.md at main · openziti-test-kitchen/zssh · GitHub
Here, the key is in the host.v1 config:

ziti edge create config "${service_name}.host.v1" host.v1 \
  '{"protocol":"tcp", "address":"localhost","port":'"${the_port}"', "listenOptions": {"bindUsingEdgeIdentity":true}}'

see the "bindUsingEdgeIdentity = true? That will allow you to have one "ssh service" and have "n" identities bind that service... Is that making sense? :slight_smile:

3

Hi @TheLumberjack,

I found this section in the documentation but I don't understand how to use the service then. So far I always configured addresses in the intercept.v1 config. Then connected via this address.
What is the effect of the listen option "listenOptions": {"bindUsingEdgeIdentity":true}?

4

Adding "bindUsingEdgeIdentity": true allows any identity that can bind the service to bind the service with a differentiator allowing dialers to dial a specific identity. so for example, if you had 100 databases in 100 locations, you could have a single service that 100 identities could bind. Then, when a client dials that service, if it specifies the identity name as the target.

This does require two things to work if you're going to use tunnelers:

  • identities are named in a way compatible with wildcard intercepts
  • the intercept uses wildcard intercepts and sets the dialOptions.identity to $dst_hostname

What that means is when you make identities, you'd make them with names like:

  • identity_one.ziti
  • identity_two.ziti
  • identity_three.ziti

or something like that, and when you make your intercept for the dialing service you would use an intercept of *.ziti... Then, when the tunneler intercepts: identity_one.ziti, it would dial the service with DialOptions.identity of "identity_one.ziti" and the service would connect to that identity.

Let me know if that's making sense.

5

Thanks for the explanation. But it doesn't seem to work for me.
I tried it with simple ssh service.
I always get a "connection refused".

This doesn't work:
host.v1

"data": {
    "address": "127.0.0.1",
    "listenOptions": {
      "bindUsingEdgeIdentity": true
    },
    "port": 22,
    "protocol": "tcp"
}

intercept.v1

"data": {
    "addresses": [
      "*.ssh.ziti"
    ],
    "dialOptions": {
      "identity": "$dst_hostname"
    },
    "portRanges": [
      {
        "high": 22,
        "low": 22
      }
    ],
    "protocols": [
      "tcp"
    ]
  }

If I try to connect via ssh I get connection refused. (ssh user@any.ssh.ziti).

The following works but doesn't provide the wanted behavior:

host.v1

"data": {
    "address": "127.0.0.1",
    "listenOptions": {
      "bindUsingEdgeIdentity": false
    },
    "port": 22,
    "protocol": "tcp"
  }

intercept.v1

"data": {
    "addresses": [
      "*.ssh.ziti"
    ],
    "portRanges": [
      {
        "high": 22,
        "low": 22
      }
    ],
    "protocols": [
      "tcp"
    ]
  }

Here I can connect with any name (ssh user@anything.ssh.ziti).

6

I just tried it to make sure it worked for me -- it did but I hit a couple bumps that might help resolve this.

Identity Must Be Named With The Intercept Suffix

the identity must be named anything.ssh.ziti - is it? For example, my test identity was aws1.ziti and this command works fine, even from windows as you can see:

ssh -i c:\users\clint\.ssh\nf\dovholuknfaws.pem ubuntu@aws1.ziti

Use All lowercase Letters

When I tried to ssh using AWS2.ziti, my tunneler would log

reason=service 5gKOWImV6nyppVNDfeh82a has no terminators for instanceId aws2.ziti

So it seems like the tunneler is taking in the hostname, intercepting it, and normalizing it down to lowercase as part of it's lookup. That means you'll need to stick to lowercase letters for this to work.

Hopefully that helps. If you need any more help tho, just let me know

7

Thanks for that information. This works!

But, since identities need to have unique names, how does it work when I bind the service to multiple identities? Do I need to set the address in the intercept for every single identity?

I figured it out.

Since there is a wildcard used in the address of the intercept

intercept address: *.ssh.ziti

all identities like the following will work:

identity1.ssh.ziti
identity2.ssh.ziti
identity3.ssh.ziti

I guess this limits the use of multiple different services on a single identity.
I my case, I would like to have a second http service as well. If it is hardcoded into the name of the identity this will not be possible?

8

Yes that's a slight limitation indeed.

If it were me, I probably would just not embed the ssh into the name and instead rely on ports to be the differentiating portion of the intercept.

  • identity1.ziti
  • identity2.ziti

or if they are some class of box that has "a web server on port 80 and ssh" then i'd use their class name:

  • webserver1.ziti
  • webserver2.ziti
9

Unfortunately it doesn't work anymore.
It seems, that after playing around with the settings in intercept and host config, it will get stuck in a not working state.
There is no possibility to for me to make it work again from the UI.
It keeps saying "Connection refused". No matter which config I am trying to use.

10

You'll need to look at the logs to understand why.

When you look in the client tunneler -- the one doing the intercepting what does it tell you? I'm going to guess it'll state "no terminators available"?

Let's look in the client tunneler and see what it reports and then go from there

11

Yes, that's the error message:

failed to connect, reason=service 4CViMbUdirtnWa7LGjpEWj has no terminators

12

Yeah, I expected with the work you're doing, you have a policy that's not working the way you want.

Here I would run policy-advisor: ziti edge policy-advisor identities -q. When you do that you'll see which identities are allowed to bind or dial a given service. Whatever service policy you had in place is no longer granting the host/binding identity the bind privilege.

Personally, this is why I like to use attributes on identities and use those attributes when making policies.

If you find your identity that should be able to bind the service and make sure that identity can bind both services, I think you'll be good to go.

13

On which device do I need to run the policy-advisor?

14

Any machine with the ziti binary installed on it -- usually I'd say for most people the easiest places is the controller, but you could run it from wherever you like.

15

I have tried multiple scenarios. It seems renaming the identity hosting the service is not a good idea. In my case it lead to a total chaos in the terminators. There will be new terminators. All of them are wrongly configured.
I have not found a way to recover from that issue.

What triggers terminators to be created?

16

There will be new terminators. All of them are wrongly configured.

What sort of chaos and what sort of wrong configuration are you referring to ? I'm not sure I understand the issue you faced?

What triggers terminators to be created?

When an identity receives a service update and the service has the 'bind' privilege it will create a terminator. This happens when the identity first authenticates or when the identity polls/receives an update for a service from the controller.

17

Terminator seem to be generated automatically. Some changes seem to trigger the generation some do not, although there is a configuration change that have an impact on terminator.

If I rename an Identity, the terminators are generated with the "old" name of the identity. The terminator has a configuration with a non existing identity. This obviously doesn't work.

18

OoooOOooooh yeah... I could see that being a problem and to be honest, I would expect that's a bug/situation we just haven't tested because it didn't occur to us. That sounds like something we probably need to test/verify/fix....

Thanks for that detail, it didn't really occur to me but it makes sense. I'll talk it over with the team and see what other people think as to if it will/should work.

19

In addition, I found out how to bind multiple services to one device without the need to have the identity named exactly like the service.

The trick is in the host config:

I have 2 identities:

identity1
identity2

There is 1 service bound to both identities.

Important is the "identity" setting in the host config:

"data": {
    "address": "127.0.0.1",
    "listenOptions": {
      "bindUsingEdgeIdentity": false,
      "identity": "$tunneler_id.name.ssh.ziti",
      "precedence": "default"
    },
    "port": 22,
    "protocol": "tcp"
  }

The part "$tunneler_id.name" is interpreted. The following ".ssh.ziti" simply added.

The intercept config:

"data": {
    "addresses": [
      "*.ssh.ziti"
    ],
    "dialOptions": {
      "identity": "$dst_hostname"
    },
    "portRanges": [
      {
        "high": 22,
        "low": 22
      }
    ],
    "protocols": [
      "tcp"
    ]
  }

This will lead to terminators with identities configured that don't really exist:

identity1.ssh.ziti
identity1.ssh.ziti

You can use them to ssh to that hosts.

This can be done for other services too:

Http:

host config:

"data": {
    "address": "127.0.0.1",
    "listenOptions": {
      "bindUsingEdgeIdentity": false,
      "identity": "$tunneler_id.name.http.ziti",
      "precedence": "default"
    },
    "port": 22,
    "protocol": "tcp"
  }

intercept:

"data": {
    "addresses": [
      "*.http.ziti"
    ],
    "dialOptions": {
      "identity": "$dst_hostname"
    },
    "portRanges": [
      {
        "high": 22,
        "low": 22
      }
    ],
    "protocols": [
      "tcp"
    ]
  }

Finally you can consume the service by using the identity name + suffix. In this case:

ssh:
identity1.ssh.ziti
identity1.ssh.ziti

http:
identity1.http.ziti
identity1.http.ziti

Hint: Don't forget about Bind and Dial policies and Service/Router policies

The only problem I am currently facing right now, is that all of a sudden, terminators are created for just one identity anymore. Maybe in a couple of hours this will work again. This seems to be very fragile.

20

Hi @Chris,
What are you using on the hosting side, ziti-edge-tunnel or a tunneler enabled edge-router? It sounds like you're hitting some bugs, and want to be sure the right person is looking into it, as those are different code bases.

Thank you,
Paul