Include Prometheus rule_files for alerting

Hi all,

We want to enable prometheus alerting and when we add the following to the generated /var/opt/gitlab/prometheus/prometheus.yml configuration file (make sure that the dir and files exist) and restart prometheus via gitlab-ctl restart prometheus it works:


  • “/var/opt/gitlab/prometheus/rules/*.rules”

However, that is not the correct place, as with a reconfigure the file will be overwritten. We can include a setting (something like prometheus[‘rule_files’] = ‘/var/opt/gitlab/prometheus/rules/*.rules’) in gitlab.rb, but apparently this is not read by the chef recipe, and indeed when we look at /opt/gitlab/embedded/cookbooks/gitlab/recipes/prometheus.rb, only specific settings are read and set on reconfigure.
So I see three options, in order of descending desirability:

  1. I’m missing something in the configuration options, but when I look at the recipe, that seems implausible
  2. We have to manually (scripted via ansible) set that after each reconfigure and restart prometheus
  3. We can copy the recipe, alter it to read the rule_files variable and make the reconfigure option read that recipe

and of course, these are just the options I see at the moment, so I’m open for alternatives ;). Any suggestions?

Kind regards