Controller metrcis - empty reply

1

How can I get the metrics from the openziti controller?

I played aroung with 0.0.0.0 and 127.0.0.1 but always get curl: (52) Empty reply from server

I'm using curl --request GET "http://127.0.0.1:10004/metrics" to querry the metrics.

My Config:

  - name: apis-metrics-localhost
    bindPoints:
      #interface - required
      # A host:port string on which network interface to listen on. 0.0.0.0 will listen on all interfaces
      - interface: 0.0.0.0:10004

        # address - required
        # The public address that external incoming requests will be able to resolve. Used in request processing and
        # response content that requires full host:port/path addresses.
        address: 0.0.0.0:10004
    options:
    apis:
      - binding: metrics
        options: {
          includeTimestamps: true
        }
2

The endpoint is a Prometheus endpoint. You can use Prometheus to collect them, or you can get them from the logs directly and parse them with something else. You do need to make sure the filters are set to record the metrics you intend to view, those configurations are on the same page.

There is also a page on using Grafana to visualize the metrics gathered by Prometheus.

3

The plan was to use netdata and catch the metrics via netdata.

The prometheus config file from netdata:

jobs:

  - name: openziti_local
    url: 'http://127.0.0.1:10004/metrics'

But I'll get following error message:

[ DEBUG ] build[manager] build.go:164 received config group ('/etc/netdata/go.d/prometheus.conf'): 1 jobs (added: 1, removed: 0)
[ DEBUG ] build[manager] build.go:313 building prometheus[openziti_local] job, config: map[__provider__:file reader __source__:/etc/netdata/go.d/prometheus.conf autodetection_retry:0 module:prometheus name:openziti_local priority:70000 update_every:10 url:http://127.0.0.1:10004/metrics]
[ ERROR ] prometheus[openziti_local] prometheus.go:113 Get "http://127.0.0.1:10004/metrics": EOF
[ ERROR ] prometheus[openziti_local] job.go:205 check failed
[ DEBUG ] run[manager] run.go:43 tick 0
4

Have you followed the authentication piece of the Prometheus page? https is the protocol, not http, so you have to use https in the url and skip TLS verification (Since it uses a private cert), or you can also set up a key and cert. Depending on how else you are securing the metrics endpoint, it's a decision to make. You have the endpoint bound to all interfaces, so if it is remotely accessible, I would secure it completely. If you are using a cloud or host based security group to keep the port closed off, then the easier thing to do is to use https and set the skip authentication (insecure_skip_verify: true).

5

Thank you. I've read this but was ignoring the https part ...

I changed the setup to

jobs:

  - name: openziti_local
    url: 'https://127.0.0.1:10004'
    tls_skip_verify: yes

with https://127.0.0.1:10004/metrics I get still EOF

[ DEBUG ] build[manager] build.go:313 building prometheus[openziti_local] job, config: map[__provider__:file reader __source__:/etc/netdata/go.d/prometheus.conf autodetection_retry:0 module:prometheus name:openziti_local priority:70000 tls_skip_verify:true update_every:10 url:https://127.0.0.1:10004/metrics]
[ ERROR ] prometheus[openziti_local] prometheus.go:113 Get "https://127.0.0.1:10004/metrics": EOF
[ ERROR ] prometheus[openziti_local] job.go:205 check failed

with https://127.0.0.1:10004 I get connection reset

[ DEBUG ] build[manager] build.go:313 building prometheus[openziti_local] job, config: map[__provider__:file reader __source__:/etc/netdata/go.d/prometheus.conf autodetection_retry:0 module:prometheus name:openziti_local priority:70000 tls_skip_verify:true update_every:10 url:https://127.0.0.1:10004]
[ ERROR ] prometheus[openziti_local] prometheus.go:113 Get "https://127.0.0.1:10004": read tcp 127.0.0.1:45088->127.0.0.1:10004: read: connection reset by peer
[ ERROR ] prometheus[openziti_local] job.go:205 check failed

Is /metrics the correct path?

I set up 0.0.0.0 only inside the docker container and mapping the container with 127.0.0.1 to the host. Also ufw is protecting the server.

6

Could be a version thing, but the syntax I have in Prometheus is insecure_skip_verify: true rather than tls_skip_verify:true. I use a remote scraper, so I have the full cert enabled, but if it helps.

My Prometheus config is

  • job_name: ziti
    scheme: https
    metrics_path: /metrics
    honor_labels: true # Ziti supplies system labels for the edge routers, so we need to obey them
    honor_timestamps: true # Honor server timestamps instead of using the scrape timestamp for metrics
    tls_config:
    cert_file: /opt/bitnami/prometheus/conf/prom-client.crt
    key_file: /opt/bitnami/prometheus/conf/prom-client.key
    insecure_skip_verify: true
    static_configs:
    • targets:
      • ':8441'
7

Thank you that help to understand.

Can you please show your ziti metric config. Does it look like:

    apis:
      - binding: metrics
        options: {
          scrapeCert: "/path/to/prom-client.crt"
        }

sudo curl -i -k --cert certs/prom-client.crt --key certs/prom-client.key https://127.0.0.1:10004/metrics
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:10004
8

Got it working. Had also a typo in the docker-compose file.
Thank you for helping with the ssl part.

9

Did you set up alerts on the openziti metrics? and if yes. Could you share them with me?

10

I never got that far on the OpenZiti side. On the CloudZiti side, we have a bunch of alerts and alarms, but that is all based on an ElasticSearch system, not Prometheus. We use Grafana for visualization of that data, but the alerts are driven with Elastalert.

11

OK. Thank you for the update.