Gitlab runner: Failed to connect to localhost port 9252 after 0 ms: Connection refused

GitLab CI/CD
, ,
1

:hugs: Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

I am trying to use Gitlab runners Datadog integration to get the Gitlab runners metrics. For that, I needed to expose the internal Prometheus metrics of the Gitlab runners. I followed this doc Monitor GitLab Runner usage | GitLab and added the listen_address field to the runners and set it to localhost:9252, but when I shell into the Gitlab runners and I curl the endpoint to get the metrics, I get the following error :

Failed to connect to localhost port 9252 after 0 ms: Connection refused

I believe this is the starting step and I can’t advance without at least getting the metrics http server. Also documentation is almost non existing when it comes to the part of integrating Datadog with Gitlab runners step by step.

Steps to reproduce

Add this global variable to the config.toml file : listen_address = "localhost:9252"
Then shell into the Gitlab runner ( setup in EKS ) and execute the following command :

curl "http://localhost:9252/metrics"

Configuration

config.toml

runners:
  config: |
    listen_address = "localhost:9252"
    [[runners]]
      name = "fargateRunners"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:20.04"
        privileged = true
        pull_policy = "if-not-present"
        allowed_pull_policies = ["always", "if-not-present"]
        image_pull_secrets = ["${docker_config_name}"]
        cpu_request = "900m"
        helper_cpu_request = "100m"
        service_account = "{{ include "gitlab-runner.fullname" . }}"
        [[runners.kubernetes.volumes.empty_dir]]
          name = "docker-certs"
          mount_path = "/certs/client"
          medium = "Memory"
        [runners.cache]
          Type = "s3"
          Path = "runner"
          Shared = true
          [runners.cache.s3]
            BucketName = "${cache_bucket_name}"
            BucketLocation = "${cache_bucket_region}"
            Insecure = false

Gitlab runners in terraform :

resource "helm_release" "gitlab_runner" {
  name       = "gitlab-runner"
  repository = "https://charts.gitlab.io"
  chart      = "gitlab-runner"
  version    = "0.50.1" # version of helm chart is 0.50.1
  namespace  = var.gitlab_release_namespace

  values = [templatefile("values.yaml",
    {
      cache_bucket_name   = aws_s3_bucket.gitlab_runner_cache.id
      cache_bucket_region = data.aws_region.current.name
      docker_config_name  = local.docker_config_name
      service_account     = aws_iam_role.gitlab_runner.arn
    }
  )]

  set_sensitive {
    name  = "runnerRegistrationToken"
    value = var.gitlab_registration_token
  }

  depends_on = [kubernetes_secret.docker_hub_config]
}

Versions

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

  • GitLab (Web: /help or self-managed system information): GitLab Enterprise Edition 16.11.0-pre
  • GitLab Runner, if self-hosted (Web /admin/runners or CLI gitlab-runner --version): 15.9.1
  • EKS: 1.27

Thanks for taking the time to be thorough in your request, it really helps! :blush: