Reduce number of Service Policies for Monitoring

1

Hi,

on the Prometheus Server and on the Client I have the linux ziti-edge-tunnel installed.
When I configure sepearte relations per Server - Client combination for host, intercept, service ans service-policies the connection is fine. But for 2 Services I do have 4 configs, 2 services and 4 policies.

Is there a way to get this easier / with less entries implemented? I tried the bindUsingEdgeIdentity example from zssh.

snip from my ansible script:

{ type: bindUsingEdgeIdentity, name: prometheusNodeSvc, protocol: "tcp", address_host: "localhost", port: 9100, bind_identity: "#prometheusClients", dial_identity: "#prometheusServer" }

ziti edge create config {{ name }}.host.v1 host.v1 "{\"protocol\":\"{{ protocol }}\",\"address\":\"{{ address_host }}\", \"port\":{{ port }}, \"listenOptions\":{\"bindUsingEdgeIdentity\":true} }"
ziti edge create service {{ name }}.svc --configs {{ name }}.host.v1
ziti edge create service-policy {{ name }}.policy.bind Bind --service-roles "@{{ name }}.svc" --identity-roles "{{ bind_identity }}"
ziti edge create service-policy {{ name }}.policy.dial Dial --service-roles "@{{ name }}.svc" --identity-roles "{{ dial_identity }}"

Tunnel Status on the Client:

"Name":"prometheusNodeSvc.svc","OwnsIntercept":true,"IsAccessible":false,"Timeout":-1,"TimeoutRemaining":-1,"Permissions":{"Bind":true,"Dial":false}

telnet localhost 9100
Trying ::1...
Connected to localhost.

Tunnel Status on the Server:

"Name":"prometheusNodeSvc.svc","OwnsIntercept":true,"IsAccessible":true,"Timeout":-1,"TimeoutRemaining":-1,"Permissions":{"Bind":false,"Dial":true}

telnet ClientIdName 9100
telnet: could not resolve ClientIdName/9100 : Temporary failure in name resolution

Did I miss something / I’m doing something wrong? Do you have a better solution to reduce the number of policies?

2

Thanks for those details. Something isn’t quite clicking for me just yet. Are you trying to have “one client” access/scrape “one prometheus server” or are you trying to do something else?

Would you just mind giving me a small overview of what it is you’re trying to actually accomplish so that I can understand better? Is it a fixed number of prometheus servers/services and a fixed number of clients?

3

I’ve around 20-30 Clients with node-exporter installed and they listen on localhost port 9100 and I have one server which polls the metrics from those 20-30 clients.

4

Good, that will make it easier for sure and it was another question I was going to ask. :wink:

Now I understand that one prometheus server will scrape 20-30 targets. You were really close with what you have. If that's the case, I would do the following.

Setup Identities for Wildcard resolution and $dst_hostname support

  • Make 30 identities, name them specifically: (this is important)
    • target1.prometheus.scrape.target
    • target2.prometheus.scrape.target
    • target3.prometheus.scrape.target
    • customer4.prometheus.scrape.target
    • whatever.prometheus.scrape.target
  • When creating the identity, give each one the attribute of "prometheus.scrape.targets" (ziti edge create identity target1.prometheus.scrape.target -a prometheus.scrape.targets)

The important point here to take note of is that they ALL end with the same ".whatever.you.want". In this case, I used ".prometheus.scrape.target" but you can use ".prom", ".prometheus", etc...

Make the Service

Now that the identities are created properly, you can make the one service you need. The important point here is that you will want to use these OpenZIti tunneler features: wildcard domains, dialOptions, listenOptions.

  • ziti edge create config "prometheus.intercept.v1" intercept.v1 '{"protocols":["tcp"],"addresses":["*.prometheus.scrape.target"], "portRanges":[{"low":9100, "high":9100}], "dialOptions":{"identity":"$dst_hostname"}}'
  • ziti edge create config "prometheus.host.v1" host.v1 '{"protocol":"tcp", "address":"127.0.0.1","port":'9100', "listenOptions": {"bindUsingEdgeIdentity":true}}'
  • ziti edge create service prometheus.svc --configs "prometheus.intercept.v1,prometheus.host.v1"

Authorize the Prometheus server and "clients"

Now the identities are created properly, they have an attribute of "prometheus.scrape.targets" and the service exists. I would update the prometheus server identity and give it the "prometheus.scrapers" attribute (referenced below). Once done, you can create the dial and bind policies:

  • ziti edge create service-policy prometheus.svc.bind Bind --service-roles "@prometheus.svc" --identity-roles "#prometheus.scrape.targets"
  • ziti edge create service-policy prometheus.svc.dial Dial --service-roles "@prometheus.svc" --identity-roles "#prometheus.scrapers"

I didn't actually test this, but i think(?) I got everything in there correct. This way, when scrape target 31 comes into the picture, all you need to do is name it "target31.prometheus.scrape.target", give it the "prometheus.scrape.targets" attribute, and deploy the ziti-edge-tunnel binary with this identity and your prometheus server will be able to now scrape it... And if you ever wanted to grant a different server scrape rights, you just give that identity the "prometheus.scrapers" attribute.

Pretty cool, right?!?! :slight_smile:

5

Testet but did not work …

Client 1:

"Name":"prometheusNodeSvc.svc","Protocols":["tcp"],"Addresses":[{"IsHost":true,"HostName":"*.intranet","Prefix":0}],"Ports":[{"High":9100,"Low":9100}],"OwnsIntercept":true,"IsAccessible":false,"Timeout":-1,"TimeoutRemaining":-1,"Permissions":{"Bind":true,"Dial":false}

Client 2:

"Name":"prometheusNodeSvc.svc","Protocols":["tcp"],"Addresses":[{"IsHost":true,"HostName":"*.intranet","Prefix":0}],"Ports":[{"High":9100,"Low":9100}],"OwnsIntercept":true,"IsAccessible":false,"Timeout":-1,"TimeoutRemaining":-1,"Permissions":{"Bind":true,"Dial":false}

On Server:

"Name":"prometheusNodeSvc.svc","Protocols":["tcp"],"Addresses":[{"IsHost":true,"HostName":"*.intranet","Prefix":0}],"Ports":[{"High":9100,"Low":9100}],"OwnsIntercept":true,"IsAccessible":true,"Timeout":-1,"TimeoutRemaining":-1,"Permissions":{"Bind":false,"Dial":true}

Telnet to Client 1:

telnet: could not resolve server.intranet/9100: Temporary failure in name resolution

Telnet to Client 2:

Trying 100.64.0.3...

Telnet to not existing DNS:

Trying 100.64.0.10...
telnet: Unable to connect to remote host: Connection refused

intercept:

{
  "name":"prometheusNodeSvc.intercept.v1",
  "data":{
    "addresses":[
      "*.intranet"],
    "dialOptions":{
      "identity":"$dst_hostname"
    },
    "portRanges":[
      {
        "high":9100,
        "low":9100
      }],
    "protocols":[
      "tcp"],
    "sourceIp":""
  }
}
6

I guess it’s time for me to build a lab. Give me a bit and I’ll try to replicate/emulate to make sure things are working

1 Like
7

Today I’ve implemented zssh and was happy, that it worked with the first server. So I started to add additional servers.

Via ssh and publickey all servers are reachable.
For zssh only the first one is reachable. For all others I get:

FATAL	error when dialing service name zssh. unable to dial service 'zssh': dial failed: service 275FjKK... has no terminators for instanceId xxx.intranet

ziti edge list service terminators 275FjKK… is only showing the working identity.

But on all servers I see

"Services":[{"Id":"275FjKK...","Name":"zssh","OwnsIntercept":true,"IsAccessible":false,"Timeout":-1,"TimeoutRemaining":-1,"Permissions":{"Bind":true,"Dial":false}},
8

Based on the GUI I see “bindUsingEdgeIdentity”:false,
But I configured it with “bindUsingEdgeIdentity”:true,

Is there a way to check this via CLI?

I deleted and recreated several times.

ziti edge create config prometheusNodeSvc.host.v1 host.v1 '{"protocol":"tcp","address":"127.0.0.1", "port":9100, "listenOptions": {"bindUsingEdgeIdentity":true}}'
9

Here is the GUI output:

{
  "name":"prometheusNodeSvc.host.v1",
  "data":{
    "address":"127.0.0.1",
    "allowedSourceAddresses":[
    ],
    "listenOptions":{
      "bindUsingEdgeIdentity":false,
      "identity":"",
      "precedence":"default"
    },
    "port":9100,
    "protocol":"tcp"
  }
}
10

CLI is schowing that bindUsingEdgeIdentity is true

ziti edge list configs -j

        {
            "_links": {
                "self": {
                    "href": "./configs/6Wb3DVDfUibkUFiouX5Jj0"
                }
            },
            "createdAt": "2023-08-19T15:16:00.433Z",
            "id": "6Wb3DVDfUibkUFiouX5Jj0",
            "tags": {},
            "updatedAt": "2023-08-19T15:16:00.433Z",
            "configType": {
                "_links": {
                    "self": {
                        "href": "./config-types/NH5p4FpGR"
                    }
                },
                "entity": "config-types",
                "id": "NH5p4FpGR",
                "name": "host.v1"
            },
            "configTypeId": "NH5p4FpGR",
            "data": {
                "address": "127.0.0.1",
                "listenOptions": {
                    "bindUsingEdgeIdentity": true
                },
                "port": 9100,
                "protocol": "tcp"
            },
            "name": "prometheusNodeSvc.host.v1"
        },

Is it possible to filter on id when using the cli?

11

About bindUsingEdgeIdentity

I’m was still trying to find time where I can make my lab. I was going to do it on Ziti TV this week but I showed BrowZer self-hosted instead. Looking at your listenOptions i see that you have it set to:

      "bindUsingEdgeIdentity":false,

That is most likely the problem. If you take a peek at the zssh readme you’ll see that for this to work you must have bindUsingEdgeIdentity to true. zssh relies on that setting, that’s how you’re able to zssh $user@${identityName}. I see then your second post state that it’s set to true but that id does look different. I suspect that’s why you’re asking about the filtering on id.

On Filtering in the CLI

The very short answer is “yes”. I’ll then get into a big long answer below to give you more info in case it’s useful…

ziti edge list identities 'id="yIutERoRPW"'

Too much detail on filtering maybe… :smiley:

With the ziti CLI you can use the -j flag. It returns json. That json ends with a small section called “filterableFields” like this:

ziti edge list identities -j | tail -22
    "meta": {
        "filterableFields": [
            "tags",
            "isSystem",
            "externalId",
            "type",
            "id",
            "updatedAt",
            "name",
            "authPolicyId",
            "isAdmin",
            "isDefaultAdmin",
            "createdAt",
            "roleAttributes"
        ],
        "pagination": {
            "limit": 10,
            "offset": 0,
            "totalCount": 3
        }
    }
}

That filterable Fields section will tell you what fields can be filtered on, and you’ll see id is one of those fields.

So for example, I have these identities:

ziti edge list identities
╭────────────┬──────────────────────────────┬────────┬────────────┬─────────────╮
│ ID         │ NAME                         │ TYPE   │ ATTRIBUTES │ AUTH-POLICY │
├────────────┼──────────────────────────────┼────────┼────────────┼─────────────┤
│ EEfGFmPm45 │ net1_offload                 │ Device │            │ Default     │
│ tlYZE7h7h  │ Default Admin                │ User   │            │ Default     │
│ yIutERoRPW │ ip-172-31-47-200-edge-router │ Router │            │ Default     │
╰────────────┴──────────────────────────────┴────────┴────────────┴─────────────╯
results: 1-3 of 3

I can filter on id like this:

ziti edge list identities 'id="yIutERoRPW"'
╭────────────┬──────────────────────────────┬────────┬────────────┬─────────────╮
│ ID         │ NAME                         │ TYPE   │ ATTRIBUTES │ AUTH-POLICY │
├────────────┼──────────────────────────────┼────────┼────────────┼─────────────┤
│ yIutERoRPW │ ip-172-31-47-200-edge-router │ Router │            │ Default     │
╰────────────┴──────────────────────────────┴────────┴────────────┴─────────────╯
results: 1-1 of 1

Next Steps

I am guessing that you have more than one config or that you might have the wrong config associated to the zssh service maybe. You can find that out by running:

ziti edge list configs
ziti edge list services -j

Note the second command adds -j. You’ll need to view the json right now as the CLI doesn’t show the configs associated to the service yet. Then find the configs block for your service:

            "configs": [
                "4LjIh2UVnHu35XPHwyFHe3",
                "KX6Nxj0uX4hCPjGQRGEiT"
            ],

Let’s make sure the correct configs are referenced.

You should be able to take that quickstart block from the readme, run it as-is, then on the sshd (server if you will) side, for those identities update them to use the attribute: #zsshSvc.binders and for the zssh “client”, update those identites with #zsshSvc.dialers and it all should “just work”.

I’ll see if I can get this running locally now… I have some time.

12

The GUI shows false. but ziti edge list configs -j show true. That's very strange for me. See also the cli output above.

I saw you BrowZer session yestarday on Ziti TV. I liked the session.

1 Like
13

Does it also only show one config? If you run ziti edge list configs what do you see?

14

zssh:

ziti edge list services 'id="2o7A4aLRZos5ekJNS9N71n"' -j
            "id": "2o7A4aLRZos5ekJNS9N71n",
            "tags": {},
            "updatedAt": "2023-08-19T16:16:26.737Z",
            "config": {},
            "configs": [
                "2SAOzk2x3fhzd5cZeC2pau"
            ],
            "encryptionRequired": true,
            "name": "zssh",
            "permissions": [
                "Bind",
                "Dial"
            ],
            "postureQueries": [],
            "roleAttributes": null,
            "terminatorStrategy": "smartrouting"



ziti edge list configs 'id="2SAOzk2x3fhzd5cZeC2pau"' -j

            "id": "2SAOzk2x3fhzd5cZeC2pau",
            "tags": {},
            "updatedAt": "2023-08-19T13:11:15.451Z",
            "configType": {
                "_links": {
                    "self": {
                        "href": "./config-types/NH5p4FpGR"
                    }
                },
                "entity": "config-types",
                "id": "NH5p4FpGR",
                "name": "host.v1"
            },
            "configTypeId": "NH5p4FpGR",
            "data": {
                "address": "localhost",
                "listenOptions": {
                    "bindUsingEdgeIdentity": true
                },
                "port": 22,
                "protocol": "tcp"
            },
            "name": "zssh.host.v1"

Actual I’ve 13 configs configured but for zssh only one. I skippt the intercept as on you example in the video.

For the monitoring service I had two configs also bindUsingEdgeIdentity true in the cli and false in the gui.

15

Ok. Thank you for confirming. 2SAOzk2x3fhzd5cZeC2pau definitely looks right to me. I’ll have to get this lab up and running. I’ll follow-up later on…

16

Oooh wait a moment. You’re using the tunneler on the prometheus side. Not zssh. I got confused !

The dial side config, did you use ‘dialOptions’? can you show me the intercept.v1 config? I bet it’s missign the dialOptions…

17

GAH. I found it above… Nevermind… Sorry, I thought I was onto something… I’ll get the lab going now :slight_smile:

18

Yes sorry. I mixed two issues in this post. But they might have the same reason.

prometheus with two configs and dialer. Posts from yesterday. zssh only with one config. Posts from today.

19

I got the lab up and running but I had success. I used docker/docker compose to emulate three sites. I put all the work out on github so that you can look at / checkout the repo and try it yourself since you can see it was working properly for me.

You can find the repo at https://github.com/dovholuknf/hello-prometheus.

The README.md explains all the steps but basically, it’s identical to what I had outlined before. I expect/hope all the commands and fields etc. should make sense (hopefully).

I narrated a (short? 10 minutes…) video showing you the steps I did (without setting up the network). You can find it here.

Hope this helps!

20

Perfect many thanks. So it should work. I used the same commands as you in the video.

So I deleted ziti-fs directory and restarted everything from scratch. Also adapted my ansible script to faster reset.

After re-enrolling all identities and adding all services I saw the no identity had a connectivity to the Edge router.

Quick and dirty I added following router policies

ziti edge create edge-router-policy my-policy --identity-roles '#all' --edge-router-roles '#all'
ziti edge create edge-router-policy my-policy --identity-roles '#all' --edge-router-roles '#public'

After that the Edge routed is connected in the GUI. But the connection is still not working. (zssh and Prometheus)

zssh error:

failure creating Dial session to service zssh

ziti edge policy-advisor identities

Service has no edge routers assigned. Adjust service edge router policies.

Will stop for today and have a deeper look tomorrow.