Issues in scraping controller metrics with otel collector prometheus receiver

Support General Questions
1

I am running controller version 1.1.15 and exposing the metrics api over localhost using its own binding.

  - name: api-metrics-localhost
    bindPoints:
      - interface: 127.0.0.1:2112
        address: 127.0.0.1:2112
    options:
    apis:
      - binding: metrics
        options: {
          includeTimestamps: true
        }

and then using otel collector prometheus receiver to scrape the metrics

receivers:
  prometheus/openziti:
    config:
      scrape_configs:
        - job_name: 'openziti-controller'
          scrape_interval: 1m
          scrape_timeout: 10s
          scheme: https
          honor_labels: true
          tls_config:
            ca_file: <CONTROLLER_CA>
            server_name: <CONTROLLER_NAME>
            insecure_skip_verify: false
          static_configs:
            - targets: ['localhost:2112']

and I have been noticing that scraping just stops after few hours ( happened 3 times until now and most recently collection stopped after 15 hours ) and otel collector reports that the target is not up.

warn        internal/transaction.go:128        Failed to scrape Prometheus endpoint        {"kind": "receiver", "name": "prometheus/openziti", "data_type": "metrics", "scrape_timestamp": 1730894266111, "target_labels": "{__name__=\"up\", instance=\"localhost:2112\", job=\"openziti-controller\"}"}

otel collector is deployed on the same AWS EC2 host as the controller.

I am not sure where the issue is. Has anyone come across this before? TIA.

2

That's certainly unexpected. I have not tried this myself and unfortunately I'm unfamiliar with "the otel collector".

Is this something you can reproduce with just the controller running or does it seem to be triggered after substantial use?

I will try to find time to test this on my own, but it'll likely be "a minute". :frowning: