Prometheus out of bounds error after reboot

I’ve installed Gitlab 15.3.3 into a Ubuntu VM, everything was doing fine after I rebooted.
Now, I’m having “out of bounds error” in Prometheus endpoint http://localhost:9187/metrics and metrics in grafana are not displayed, none of them. Anyone knows of to restart the service? I’ve already tried sudo gitlab-ctl restart and sudo gitlab-ctl reconfigure several times.

sudo gitlab-ctl status seems fine too,
run: alertmanager: (pid 17900) 2298s; run: log: (pid 1041) 0s
run: gitaly: (pid 17909) 2298s; run: log: (pid 1011) 0s
run: gitlab-exporter: (pid 17922) 2297s; run: log: (pid 1013) 0s
run: gitlab-kas: (pid 17925) 2297s; run: log: (pid 1033) 0s
run: gitlab-workhorse: (pid 17937) 2296s; run: log: (pid 1042) 0s
run: grafana: (pid 17946) 2296s; run: log: (pid 1036) 0s
run: logrotate: (pid 17961) 2295s; run: log: (pid 1021) 0s
run: nginx: (pid 17970) 2295s; run: log: (pid 1035) 0s
run: node-exporter: (pid 17976) 2295s; run: log: (pid 1016) 0s
run: postgres-exporter: (pid 18058) 2294s; run: log: (pid 1018) 0s
run: postgresql: (pid 18066) 2294s; run: log: (pid 1017) 0s
run: prometheus: (pid 18079) 2293s; run: log: (pid 1014) 0s
run: puma: (pid 18089) 2292s; run: log: (pid 1025) 0s
run: redis: (pid 18094) 2292s; run: log: (pid 1023) 0s
run: redis-exporter: (pid 18101) 2292s; run: log: (pid 1015) 0s
run: sidekiq: (pid 18109) 2290s; run: log: (pid 1019) 0s

Here is the log:
==> /var/log/gitlab/prometheus/current <==
2022-09-20_03:43:56.23501 level=warn ts=2022-09-20T03:43:56.234Z caller=scrape.go:1534 component=“scrape manager” scrape_pool=postgres target=http://localhost:9187/metrics msg=“Error on ingesting samples that are too old or are too far into the future” num_dropped=1405
2022-09-20_03:43:56.23515 level=warn ts=2022-09-20T03:43:56.235Z caller=scrape.go:1280 component=“scrape manager” scrape_pool=postgres target=http://localhost:9187/metrics msg=“Append failed” err=“out of bounds”
2022-09-20_03:43:56.23519 level=warn ts=2022-09-20T03:43:56.235Z caller=scrape.go:1229 component=“scrape manager” scrape_pool=postgres target=http://localhost:9187/metrics msg=“Appending scrape report failed” err=“out of bounds”
2022-09-20_03:43:56.91672 level=warn ts=2022-09-20T03:43:56.916Z caller=scrape.go:1534 component=“scrape manager” scrape_pool=gitlab-sidekiq target=http://127.0.0.1:8082/metrics msg=“Error on ingesting samples that are too old or are too far into the future” num_dropped=5572
2022-09-20_03:43:56.91715 level=warn ts=2022-09-20T03:43:56.917Z caller=scrape.go:1229 component=“scrape manager” scrape_pool=gitlab-sidekiq target=http://127.0.0.1:8082/metrics msg=“Appending scrape report failed” err=“out of bounds”

This solved the issue so far, in case anyone else need it:

  1. sudo gitlab-ctl restart
  2. cd /var/opt/gitlab/prometheus/data
  3. rm -rf 0* wal/0* wal/checkpoint.0*
1 Like

rm -rf 0* wal/0* wal/checkpoint.0*

it works!

thx