Blank Content-Type Version in Ziti Controller Metrics

1

Hey guys,

We're experiencing some issues with ziti controller metrics. Prometheus experiences following error:

Error scraping target: non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target

"No Fallback Protocol" can be ignored here but the blank Content-Type (missing the version) is the problem. The header of the controllers metrics looks like this:

< HTTP/1.1 200 OK
< Date: Fri, 20 Dec 2024 12:39:03 GMT
< Content-Type: text/plain; charset=utf-8
< Transfer-Encoding: chunked

After that the metrics begin. In the Content-Type it misses the "version=0.0.4" entry. For example following some metrics by node exporter:

< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Fri, 20 Dec 2024 12:38:17 GMT
< Transfer-Encoding: chunked

The metrics bind in the controller yaml looks like in the documentary:

  - name: Metrics
    bindPoints:
      - interface: 0.0.0.0:2112
        address: 0.0.0.0:2112
    options:
    apis:
      - binding: metrics
        options: {
          includeTimestamps: true
        }

The prometheus config looks like this:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'ziti'
    scheme: https
    metrics_path: /metrics
    honor_labels: true
    honor_timestamps: true
    tls_config:
      insecure_skip_verify: true
    static_configs:
      - targets: ['172.17.0.1:2112']

How can this be fixed or am I missing any point here? Tried older and newest ziti sdks and older and newest prometheus docker image. Any ideas are welcome!

2

Hi @Misc, thanks for starting a discussion on this topic. Is this a new issue that occurred after you upgraded your controller? Did anything relevant change recently or are you trying to just set up prometheus monitoring for the first time?

3

It's a new issue but I cant reproduce when it occured. We had the same setup running now for a pretty long while. What I experienced before is, that sometimes the prometheus container needs one or two rebuilds until it can scrape the controllers metrics but after a few tries it worked and pretty long and well too so I never looked deeper if there are any problems. Today I got the info of missing datas in our grafana dashboards so I looked into it and ran into that error. Checked it with some other productive and testing systems with all kinds of versions and it seems like its running nowhere.

4

I found a very old prometheus docker image that works with the config. It seems like the newer version enforce the right header or something like this. But that's just a temporary fix.

For those who may have the same problem: prometheus 2.51.2 seems to work.

5

Ok so it sounds to me like we probably need to update the scrape target to work with newer Prometheus builds. I'll file an issue. What version were you trying to run? "latest"?

6

Yeah I tried the latest Images, with these it didnt work

7

Thanks, I've filled this issue

https://github.com/openziti/ziti/issues/2608