[SOLVED] Gitlab Docker prometheus_multiproc_dir and /-/metrics

Hello all,

I just tried to enable the Prometheus metrics on /-/metrics as per https://docs.gitlab.com/ce/administration/monitoring/prometheus/gitlab_metrics.html.

I run Gitlab 9.5 from a Docker image. The settings has a check box to enable these metrics, but states that the “Environment variable prometheus_multiproc_dir does not exist or is not pointing to a valid directory.”.

I stopped, removed and restarted the Docker container using -e prometheus_multiproc_dir=/tmp - this seemed to have 0 effect; the warning persists. I tried a number of different directories; warning still persists. I logged in to the gitlab container and echo’d out $prometheus_multiproc_dir and it shows the directory I set. The environment variable is definitely there.

When I access /-/metrics, I get a 404. This might be because nothing is running to provide the metrics as even with the checkbox on, it seems to do nothing as long as this warning persists.

What am I missing here? How am I supposed to set this environment variable? I also tried to set it in gitlab_rails[‘prometheus_multiproc_dir’] - also to no avail.

Thanks!

1 Like

I’m having the same problem.

I’ve tried with

echo “export prometheus_multiproc_dir=/dev/shm” >> /etc/profile.d/prometheus.sh

Made the script executable, tried with user git and git-prometheus, all of them could read the env var but the settings panel still showed the error.

I’m using CentOS 6 on openvz.

I just set "config.multiprocess_files_dir = ‘/tmp’ in /opt/gitlab/embedded/service/gitlab-rails/config/initializers/7_promethesus_metrics.rb and did a restart without doing a reconfigure.

Hi @Fahr, looks like the prometheus_multiproc_dir should be added as an ENV variable for gitlab. When working in docker using docker-compose these options can be set in the docker-compose.yml file like this:

  environment:
    GITLAB_OMNIBUS_CONFIG: |
      gitlab_rails['env'] = { 'prometheus_multiproc_dir' => '/dev/shm' }

I faced the same issue yesterday and finally found the fix here

Make sure to restart unicorn and clear all the old metrics

More information on setting environment variables.

2 Likes

@shehzi - thanks! This option actually worked. I think some documentation to this end might be in place; far too many places to set all sorts of variables now… especially for Docker.

@Fahr I definitely agree, finding that configuration was a wild goose chase for me too!

Do mark the issue as solved so that others may find it as well.