Monitor GitLab on an external Prometheus Server

Observability
1

Hi!

On my self-hosted GitLab instance, I can monitor these metrics exposed with this URL:

https://mygitlab.com/-/metrics?token=mytoken

I have a separate external Prometheus server running, and I would like to add these metrics on it.

In the prometheus.yml file, I added this:

  - job_name: "gitlab"
    metrics_path: "/-/metrics"
    scheme: https
    scrape_interval: "5s"
    authorization:
        type: Bearer
        credentials: mytoken
        tls_config:
          insecure_skip_verify: true
    static_configs:
      - targets:
          - "mygitlab.com"
        labels:
          name: "GitLab"

and restarted Prometheus.

It does not work. In the UI of Prometheus, it says ‘server returned HTTP status 404 Not Found’ for this endpoint.

Is there something wrong in my job configuration?

2

The docs mention that you need to explicitly allow the IP address of the external Prometheus server to access the GitLab /metrics endpoint. Suggest checking if set.

3

Thank you for the answer @dnsmichi. I explicitly allowed the IP address of the VM where Prometheus runs and nothing changes.

I also ran curl https://mygitlab.com/-/metrics?token=mytoken on the terminal of this VM and nothing happens as well, although it works when I put the URL in my browser.

4

I assume I need to change some network settings in order for this to work, but I fail to see which exactly. Port 443 is open between the two Vms.