Clear up Postges / Prometheus data

Observability
1

I recently inherited a GitLab Omnibus instance in a Docker container.
Its getting tight with space on there. So tight that I cant even create backups.

ncdu tells me that /var/opt/gitlab/prometheus/data/wal/ has about 15Gig of Prometheus data on there.
the cleanup docks doesnt mention Prometheus at all.
Can I get rid of it in a gentle way?

Thanks for taking the time to help here, it really matters! :blush:

2

After sinking in another couple of hours I might got somewhere.

adding following to my staging gitlab.rb clears up the data qite nicely.

prometheus['enable'] = true
postgresql['min_wal_size'] = "50MB"
postgresql['max_wal_size'] = "100MB"
prometheus['flags'] = {
  'storage.tsdb.path' => "/var/opt/gitlab/prometheus/data",
  'storage.tsdb.retention.time' => "60s",
  'config.file' => "/var/opt/gitlab/prometheus/prometheus.yml",
}

Actualy I would like to have 'web.enable-admin-api'in that block too - but the reconfigure passes me an = into when i use an empty qoute, wich gets the instance into trouble.

these settings give you an /opt/gitlab/service/prometheus/runlike this

#!/bin/sh
exec 2>&1

umask 077
exec chpst -P -e /opt/gitlab/etc/prometheus/env \
  -U gitlab-prometheus:gitlab-prometheus \
  -u gitlab-prometheus:gitlab-prometheus \
  /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitlab/prometheus/prometheus.yml --storage.tsdb.retention.time=60s

note:
when I would be able to pass the web.enable-admin-api flag I would run

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={_name__=~".+"}'
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'

to get rid of all of the series

Now I have to get this only into an docker-compose.yaml enviroment…

1 Like
3

After editing the gitlab.rb file you need to run “sudo gitlab-ctl reconfigure” to free up space. A simple gitlab restart won’t do. Btw I only made use of this change and the /var/opt/gitlab/prometheus directory went down from 8.3GB to 11MB.