# Controller metrcis - empty reply **URL:** \ **Category:** Uncategorized\ **Created:** [September 4, 2023, 10:22am UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597 "2023-09-04T10:22:48Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Metz](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Metz](https://openziti.discourse.group/u/Metz)\ **Post date:** [September 4, 2023, 10:22am UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/1 "2023-09-04T10:22:48Z")
How can I get the metrics from the openziti controller? I played aroung with 0.0.0.0 and 127.0.0.1 but always get `curl: (52) Empty reply from server` I'm using `curl --request GET "http://127.0.0.1:10004/metrics"` to querry the metrics. My Config: ```auto - name: apis-metrics-localhost bindPoints: #interface - required # A host:port string on which network interface to listen on. 0.0.0.0 will listen on all interfaces - interface: 0.0.0.0:10004 # address - required # The public address that external incoming requests will be able to resolve. Used in request processing and # response content that requires full host:port/path addresses. address: 0.0.0.0:10004 options: apis: - binding: metrics options: { includeTimestamps: true } ``` ---
**Author:** ![mike.gorman](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/mike.gorman/32/7_2.png) [@mike.gorman](https://openziti.discourse.group/u/mike.gorman)\ **Post date:** [September 4, 2023, 10:38am UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/2 "2023-09-04T10:38:06Z")
The endpoint is a Prometheus endpoint. You can use Prometheus to collect them, or you can get them from the logs directly and parse them with something else. You do need to make sure the filters are set to record the metrics you intend to view, those configurations are on the same page. > **[Prometheus Endpoint | OpenZiti](https://openziti.io/docs/learn/core-concepts/metrics/prometheus/)** > > The Ziti Controller can expose a /metrics endpoint that serves network metrics in the Prometheus text exposition format. There is also a page on using Grafana to visualize the metrics gathered by Prometheus. > **[Using Grafana | OpenZiti](https://openziti.io/docs/learn/core-concepts/metrics/grafana/)** > > Grafana has a marvelous datasource type called Infinity. It is highly flexible in it's own right, and the numerous options it provides to configure additional funcitonality makes it a great choice for interacting with APIs for various purposes. In... ---
**Author:** ![Metz](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Metz](https://openziti.discourse.group/u/Metz)\ **Post date:** [September 4, 2023, 12:34pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/3 "2023-09-04T12:34:00Z")
The plan was to use netdata and catch the metrics via netdata. The prometheus config file from netdata: ```auto jobs: - name: openziti_local url: 'http://127.0.0.1:10004/metrics' ``` But I'll get following error message: ```auto [DEBUG] build[manager] build.go:164 received config group ('/etc/netdata/go.d/prometheus.conf'): 1 jobs (added: 1, removed: 0) [DEBUG] build[manager] build.go:313 building prometheus[openziti_local] job, config: map[__provider__ :file reader __source__ :/etc/netdata/go.d/prometheus.conf autodetection_retry:0 module:prometheus name:openziti_local priority:70000 update_every:10 url:http://127.0.0.1:10004/metrics] [ERROR] prometheus[openziti_local] prometheus.go:113 Get "http://127.0.0.1:10004/metrics": EOF [ERROR] prometheus[openziti_local] job.go:205 check failed [DEBUG] run[manager] run.go:43 tick 0 ``` ---
**Author:** ![mike.gorman](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/mike.gorman/32/7_2.png) [@mike.gorman](https://openziti.discourse.group/u/mike.gorman)\ **Post date:** [September 4, 2023, 12:58pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/4 "2023-09-04T12:58:44Z")
Have you followed the authentication piece of the Prometheus page? https is the protocol, not http, so you have to use https in the url and skip TLS verification (Since it uses a private cert), or you can also set up a key and cert. Depending on how else you are securing the metrics endpoint, it's a decision to make. You have the endpoint bound to all interfaces, so if it is remotely accessible, I would secure it completely. If you are using a cloud or host based security group to keep the port closed off, then the easier thing to do is to use https and set the skip authentication (insecure\_skip\_verify: true). ---
**Author:** ![Metz](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Metz](https://openziti.discourse.group/u/Metz)\ **Post date:** [September 4, 2023, 1:20pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/5 "2023-09-04T13:20:57Z")
Thank you. I've read this but was ignoring the https part ... I changed the setup to ```auto jobs: - name: openziti_local url: 'https://127.0.0.1:10004' tls_skip_verify: yes ``` with `https://127.0.0.1:10004/metrics` I get still EOF ```auto [DEBUG] build[manager] build.go:313 building prometheus[openziti_local] job, config: map[__provider__ :file reader __source__ :/etc/netdata/go.d/prometheus.conf autodetection_retry:0 module:prometheus name:openziti_local priority:70000 tls_skip_verify:true update_every:10 url:https://127.0.0.1:10004/metrics] [ERROR] prometheus[openziti_local] prometheus.go:113 Get "https://127.0.0.1:10004/metrics": EOF [ERROR] prometheus[openziti_local] job.go:205 check failed ``` with `https://127.0.0.1:10004` I get connection reset ```auto [DEBUG] build[manager] build.go:313 building prometheus[openziti_local] job, config: map[__provider__ :file reader __source__ :/etc/netdata/go.d/prometheus.conf autodetection_retry:0 module:prometheus name:openziti_local priority:70000 tls_skip_verify:true update_every:10 url:https://127.0.0.1:10004] [ERROR] prometheus[openziti_local] prometheus.go:113 Get "https://127.0.0.1:10004": read tcp 127.0.0.1:45088->127.0.0.1:10004: read: connection reset by peer [ERROR] prometheus[openziti_local] job.go:205 check failed ``` Is `/metrics` the correct path? I set up 0.0.0.0 only inside the docker container and mapping the container with 127.0.0.1 to the host. Also ufw is protecting the server. ---
**Author:** ![mike.gorman](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/mike.gorman/32/7_2.png) [@mike.gorman](https://openziti.discourse.group/u/mike.gorman)\ **Post date:** [September 4, 2023, 1:37pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/6 "2023-09-04T13:37:57Z")
Could be a version thing, but the syntax I have in Prometheus is insecure\_skip\_verify: true rather than tls\_skip\_verify:true. I use a remote scraper, so I have the full cert enabled, but if it helps. My Prometheus config is > - job\_name: ziti > scheme: https > metrics\_path: /metrics > honor\_labels: true # Ziti supplies system labels for the edge routers, so we need to obey them > honor\_timestamps: true # Honor server timestamps instead of using the scrape timestamp for metrics > tls\_config: > cert\_file: /opt/bitnami/prometheus/conf/prom-client.crt > key\_file: /opt/bitnami/prometheus/conf/prom-client.key > insecure\_skip\_verify: true > static\_configs: > - targets: > - ':8441' ---
**Author:** ![Metz](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Metz](https://openziti.discourse.group/u/Metz)\ **Post date:** [September 4, 2023, 2:13pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/7 "2023-09-04T14:13:22Z")
Thank you that help to understand. Can you please show your ziti metric config. Does it look like: ```auto apis: - binding: metrics options: { scrapeCert: "/path/to/prom-client.crt" } ``` ```auto sudo curl -i -k --cert certs/prom-client.crt --key certs/prom-client.key https://127.0.0.1:10004/metrics curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:10004 ``` ---
**Author:** ![Metz](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Metz](https://openziti.discourse.group/u/Metz)\ **Post date:** [September 4, 2023, 2:19pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/8 "2023-09-04T14:19:36Z")
Got it working. Had also a typo in the docker-compose file. Thank you for helping with the ssl part. ---
**Author:** ![Metz](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Metz](https://openziti.discourse.group/u/Metz)\ **Post date:** [September 4, 2023, 4:23pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/9 "2023-09-04T16:23:23Z")
Did you set up alerts on the openziti metrics? and if yes. Could you share them with me? ---
**Author:** ![mike.gorman](https://yyz2.discourse-cdn.com/free1/user_avatar/openziti.discourse.group/mike.gorman/32/7_2.png) [@mike.gorman](https://openziti.discourse.group/u/mike.gorman)\ **Post date:** [September 4, 2023, 6:32pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/10 "2023-09-04T18:32:37Z")
I never got that far on the OpenZiti side. On the CloudZiti side, we have a bunch of alerts and alarms, but that is all based on an ElasticSearch system, not Prometheus. We use Grafana for visualization of that data, but the alerts are driven with Elastalert. ---
**Author:** ![Metz](https://avatars.discourse-cdn.com/v4/letter/m/c77e96/32.png) [@Metz](https://openziti.discourse.group/u/Metz)\ **Post date:** [September 4, 2023, 6:37pm UTC](https://openziti.discourse.group/t/controller-metrcis-empty-reply/1597/11 "2023-09-04T18:37:07Z")
OK. Thank you for the update.