GitLab monitoring with Prometheus and Datadog

Observability
1

I’m trying to set up monitoring for GitLab and my runners. My company doesn’t use Prometheus, but it does use Datadog, so I’m trying to work with that. This might be more of a Datadog problem, but I’m posting here in case anyone has experience with this integration. (By the way, is this a silly thing to do? It doesn’t seem well-supported.)

Here’s where I’m at:

GitLab

I turned on Prometheus in the GitLab UI and restarted GitLab. I added the GitLab integration for Datadog with curl -o /etc/dd-agent/checks.d/gitlab.py https://raw.githubusercontent.com/DataDog/integrations-core/master/gitlab/datadog_checks/gitlab/gitlab.py, and I used the default gitlab.yaml file. Datadog says:

    gitlab (custom)
    ---------------
      - instance #0 [ERROR]: "('U', 'n', 's', 'u', 'p', 'p', 'o', 'r', 't', 'e', 'd', ' ', 'c', 'o', 'n', 't', 'e', 'n', 't', '-', 't', 'y', 'p', 'e', ' ', 'p', 'r', 'o', 'v', 'i', 'd', 'e', 'd', ':', ' ', 't', 'e', 'x', 't', '/', 'h', 't', 'm', 'l', ';', ' ', 'c', 'h', 'a', 'r', 's', 'e', 't', '=', 'u', 't', 'f', '-', '8')"
      - Collected 0 metrics, 0 events & 0 service checks

Why is this happening? How am I supposed to fix it?

GitLab runner

I added metrics_server = "localhost:9252" to my /etc/gitlab-runner/config.toml. Cool, now curling localhost:9252/metrics and the output looks like Prometheus metrics.

I added the gitlab-runner integration with curl -o /etc/dd-agent/checks.d/gitlab_runner.py https://raw.githubusercontent.com/DataDog/integrations-core/master/gitlab_runner/datadog_checks/gitlab_runner/gitlab_runner.py, and I used the default gitlab_runner.yaml file. Datadog says:

    gitlab_runner (custom)
    ----------------------
      - instance #0 [OK]
      - Collected 78 metrics, 0 events & 1 service check

… But I don’t see how to add monitoring for this on the Datadog UI. The integrations section doesn’t have anything listed for Prometheus or GitLab. How do I do this?

2

I figured it out. Basically Datadog issues, not GitLab issues.

To monitor the metrics collected:

  1. Go to the New Monitor page in Datadog.
  2. Click “Metric” under “Select a monitor type.”
  3. In section 2, “Define the metric”, there’s a field for “Get”. Enter “gitlab” and you’ll see all the metrics for GitLab and GitLab Runner.

To fix the “unsupported content type” thing with GitLab monitoring:

  1. prometheus_endpoint in the default config file is missing /metrics. Should be prometheus_endpoint: http://localhost:9090/metrics.