Modify bundled scrape_config (add external node to 'node' job)

Observability
1

Our current gitlab installation is split on two servers:

  • one with all services (including prometheus) except the db
  • one other with the postgres db.

The postgres server is currently holding a classic postgres db installed through the official postgresql yum repository.

We have prometheus and grafana enabled on the main server and can take advantage of the dashboards. But with such a setup, we don’t get any metrics back from the DB.

For this reason and also for the future ease on maintenance (upgrades, etc…), I would like to switch the install on the db server to use the omnibus install.

I have done most of the job: configuring omnibus to install db only, enable the necessary exporters on the server… But I’m stuck when it comes to modifying the scrape_config on the main server holding prometheus. Here is the relevant part of the config in gitlab.rb on that server.

prometheus['scrape_configs'] = [
  {
    'job_name': 'postgres',
    'static_configs' => [
     'targets' => ['my.postgres.server.ip:9187'],
    ],
  },
  {
    'job_name': 'node',
    'static_configs' => [
        'targets' => ['localhost:9100', 'my.postgres.server.ip:9100']
    ]
  },
]

Installed alone, the postgres job ‘works’ (I have a problem with data but I think I can sort it out…). My real concern is with the node job. As is, it simply crashes prometheus on start complaining that job with the same name already exists. This is the exact message in /var/log/gitlab/prometheus/current

2020-07-10_14:03:30.54499 level=info ts=2020-07-10T14:03:30.528Z caller=main.go:747 msg=“Loading configuration file” filename=/var/opt/gitlab/prometheus/prometheus.yml
2020-07-10_14:03:30.55467 level=error ts=2020-07-10T14:03:30.554Z caller=main.go:587 msg=“Error reloading config” err=“couldn’t load configuration (–config.file="/var/opt/gitlab/prometheus/prometheus.yml"): parsing YAML file /var/opt/gitlab/prometheus/prometheus.yml: found multiple scrape configs with job name "node"”

And with no surprise this is exactly what I find in the generated prometheus config file

scrape_configs:
# [...]
- job_name: node
  static_configs:
  - targets:
    - 0.0.0.0:9100
# [...]
- job_name: node
  static_configs:
  - targets:
    - localhost:9100
    - 172.28.128.21:9100

What I was expecting is that my custom config would replace the stock one, or that gitlab would merge both. My requirement is simply to add a target to the one gitlab creates on its own on the node job. I did not find any documentation on how to do that: is it possible ?

An other option would be to ask gitlab not to generate the stock scrape configurations and manage them A-Z myself but I did not see how to do that except by disabling prometheus entirely (which kind of misses the point…)

I know that I can deploy prometheus + grafana on a separate dedicated server and fully manage the scrape configuration myself. But I need to keep this on the main server for time being.

Any clue greatly appreciated (even experimental ones !). Thanks.