Gitlab-ctl status/reconfigure/ and gitlab-rake freezes more than 24h. Multiple gitlab versions

How to Use GitLab
1

Hello =]

Problem to solve

Why gitlab could not start correctly after shutting down ubuntu vm few times?

I’ve done it 4 times on fresh vms with different ubuntu versions. How can I make gitlab server running after this?

My main server got sth like that. I’ve tried imo all ideas I could find in net, but without success. Maybe I’m missing sth.

Steps to reproduce

  1. VM(hyper-V) 8-16 cores, 4-20GB RAM or dynamic, 1TB drive. I have tired a lot of configuration of vm.
  2. Ubuntu 20,22,24 does not matter
  3. Gitlab 14 or 17, tried a few versions
  4. Simple setup like here: Download and install GitLab | GitLab (without postfix)
  5. Wait till it is running.
  6. Restart/shutdown vm or host in gui or in cmd
  7. Repeat steps 5 and 6 5/10 times different ways:
  8. Gitlab-ctl is not responding. 502 frontend.

Configuration

Fresh install with only EXTERNAL_URL changed.

Versions

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

  • 14.9.5 and 17.x.x
Fresh gitlab runsvdir log 
● gitlab-runsvdir.service - GitLab Runit supervision process
     Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-12-03 09:41:42 CET; 5min ago
   Main PID: 34071 (runsvdir)
      Tasks: 340 (limit: 4915)
     Memory: 3.8G (peak: 3.8G)
        CPU: 1min 53.500s
     CGroup: /system.slice/gitlab-runsvdir.service
             ├─34071 runsvdir -P /opt/gitlab/service "log: ..........................................................................................................................................................................................................................................................................................................................................................................................................."
             ├─34096 runsv logrotate
             ├─34098 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
             ├─34102 sleep 600
             ├─34106 svlogd -tt /var/log/gitlab/logrotate
             ├─34131 runsv redis
             ├─34133 "/opt/gitlab/embedded/bin/redis-server unixsocket:/var/opt/gitlab/redis/redis.socket"
             ├─34142 svlogd -tt /var/log/gitlab/redis
             ├─34152 runsv gitaly
             ├─34154 /opt/gitlab/embedded/bin/gitaly-wrapper /opt/gitlab/embedded/bin/gitaly serve /var/opt/gitlab/gitaly/config.toml
             ├─34160 /opt/gitlab/embedded/bin/gitaly serve /var/opt/gitlab/gitaly/config.toml
             ├─34180 svlogd /var/log/gitlab/gitaly
             ├─34237 runsv postgresql
             ├─34239 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
             ├─34243 "postgres: checkpointer "
             ├─34244 "postgres: background writer "
             ├─34245 "postgres: walwriter "
             ├─34246 "postgres: autovacuum launcher "
             ├─34247 "postgres: stats collector "
             ├─34248 "postgres: logical replication launcher "
             ├─34281 svlogd -tt /var/log/gitlab/postgresql
             ├─34419 runsv gitlab-kas
             ├─34421 /opt/gitlab/embedded/bin/gitlab-kas --configuration-file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml
             ├─34439 svlogd -tt /var/log/gitlab/gitlab-kas
             ├─34501 runsv crond
             ├─34503 /opt/gitlab/embedded/bin/go-crond --include=/var/opt/gitlab/crond
             ├─34513 svlogd -tt /var/log/gitlab/crond
             ├─34516 runsv puma
             ├─34518 "puma 6.4.3 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,tcp://127.0.0.1:8080) [gitlab-puma-worker]"
             ├─34526 svlogd -tt /var/log/gitlab/puma
             ├─34540 runsv sidekiq
             ├─34542 ruby /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster -e production -r /opt/gitlab/embedded/service/gitlab-rails -c 20 --timeout 25 "*"
             ├─34550 svlogd /var/log/gitlab/sidekiq
             ├─34551 "sidekiq 7.2.4 queues:default,mailers [0 of 20 busy]"
             ├─34553 sidekiq_exporter
             ├─34566 runsv gitlab-workhorse
             ├─34596 runsv nginx
             ├─34599 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/sockets/socket -authBackend http://localhost:8080 -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr "" -prometheusListenAddr localhost:9229 -secretPath /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret -logFormat json -config config.toml
             ├─34610 svlogd /var/log/gitlab/gitlab-workhorse
             ├─34632 svlogd -tt /var/log/gitlab/nginx
             ├─34680 "puma: cluster worker 0: 34518 [gitlab-puma-worker]"
             ├─34682 "puma: cluster worker 1: 34518 [gitlab-puma-worker]"
             ├─34684 "puma: cluster worker 2: 34518 [gitlab-puma-worker]"
             ├─34686 "puma: cluster worker 3: 34518 [gitlab-puma-worker]"
             ├─34688 "puma: cluster worker 4: 34518 [gitlab-puma-worker]"
             ├─34690 "puma: cluster worker 5: 34518 [gitlab-puma-worker]"
             ├─34692 "puma: cluster worker 6: 34518 [gitlab-puma-worker]"
             ├─34694 "puma: cluster worker 7: 34518 [gitlab-puma-worker]"
             ├─34815 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34845 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34846 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34847 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34870 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34871 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34872 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34873 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34874 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34875 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34876 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34877 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34878 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34879 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34880 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34881 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34882 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34883 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34884 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34885 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34886 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34887 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34888 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34889 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34890 "postgres: gitlab gitlabhq_production [local] idle"
             ├─34891 "postgres: gitlab gitlabhq_production [local] idle"
             ├─35126 runsv registry
             ├─35128 /opt/gitlab/embedded/bin/registry serve ./config.yml
             ├─35143 svlogd -tt /var/log/gitlab/registry
             ├─35160 "nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx"
             ├─35192 "nginx: worker process"
             ├─35193 "nginx: worker process"
             ├─35194 "nginx: worker process"
             ├─35195 "nginx: worker process"
             ├─35196 "nginx: worker process"
             ├─35197 "nginx: worker process"
             ├─35198 "nginx: worker process"
             ├─35199 "nginx: worker process"
             ├─35200 "nginx: cache manager process"
             ├─35201 "nginx: cache loader process"
             ├─35225 "postgres: gitlab gitlabhq_production [local] idle"
             ├─35226 "postgres: gitlab gitlabhq_production [local] idle"
             ├─35229 runsv node-exporter
             ├─35231 /opt/gitlab/embedded/bin/node_exporter --web.listen-address=localhost:9100 --collector.mountstats --collector.runit --collector.runit.servicedir=/opt/gitlab/sv --collector.textfile.directory=/var/opt/gitlab/node-exporter/textfile_collector
             ├─35240 svlogd -tt /var/log/gitlab/node-exporter
             ├─35249 runsv gitlab-exporter
             ├─35251 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/gitlab-exporter web -c /var/opt/gitlab/gitlab-exporter/gitlab-exporter.yml
             ├─35262 svlogd -tt /var/log/gitlab/gitlab-exporter
             ├─35265 runsv redis-exporter
             ├─35266 svlogd -tt /var/log/gitlab/redis-exporter
             └─35267 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///var/opt/gitlab/redis/redis.socket

Dec 03 09:41:42 [domain] systemd[1]: Started gitlab-runsvdir.service - GitLab Runit supervision process.
After few shutdowns/restarts

● gitlab-runsvdir.service - GitLab Runit supervision process
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-12-03 10:46:46 CET; 5min ago
Main PID: 2370 (runsvdir)
Tasks: 174 (limit: 4915)
Memory: 1.2G (peak: 1.2G)
CPU: 28.039s
CGroup: /system.slice/gitlab-runsvdir.service
├─2370 runsvdir -P /opt/gitlab/service “log: /gitlab/service: warning: unable to read directory /opt/gitlab/service: invalid argument\nrunsvdir /opt/gitlab/service: warning: unable to read directory /opt/gitlab/service: invalid argument\nrunsvdir /opt/gitlab/service: warning: unable to read directory /opt/gitlab/service: invalid argument\nrunsvdir /opt/gitlab/service: warning: unable to read directory /opt/gitlab/service: invalid argument\n”
├─2376 runsv gitlab-kas
├─2377 runsv crond
├─2378 runsv postgresql
├─2379 runsv gitaly
├─2380 runsv postgres-exporter
├─2381 runsv redis-exporter
├─2382 runsv redis
├─2383 runsv gitlab-exporter
├─2384 runsv alertmanager
├─2385 runsv nginx
├─2386 runsv sidekiq
├─2387 runsv registry
├─2388 runsv gitlab-workhorse
├─2389 runsv node-exporter
├─2390 runsv prometheus
├─2391 runsv logrotate
├─2392 runsv puma
├─2393 svlogd -tt /var/log/gitlab/postgres-exporter
├─2394 svlogd -tt /var/log/gitlab/registry
├─2395 svlogd -tt /var/log/gitlab/alertmanager
├─2396 svlogd -tt /var/log/gitlab/node-exporter
├─2397 svlogd /var/log/gitlab/gitaly
├─2398 svlogd /var/log/gitlab/gitlab-workhorse
├─2399 svlogd -tt /var/log/gitlab/nginx
├─2400 svlogd -tt /var/log/gitlab/crond
├─2401 svlogd /var/log/gitlab/sidekiq
├─2402 svlogd -tt /var/log/gitlab/puma
├─2403 /opt/gitlab/embedded/bin/go-crond --include=/var/opt/gitlab/crond
├─2404 ruby /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster -e production -r /opt/gitlab/embedded/service/gitlab-rails -c 20 --timeout 25 “*”
├─2405 /opt/gitlab/embedded/bin/node_exporter --web.listen-address=localhost:9100 --collector.mountstats --collector.runit --collector.runit.servicedir=/opt/gitlab/sv --collector.textfile.directory=/var/opt/gitlab/node-exporter/textfile_collector
├─2406 “nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx”
├─2407 ruby /opt/gitlab/embedded/bin/bundle exec puma -C /var/opt/gitlab/gitlab-rails/etc/puma.rb
├─2408 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/sockets/socket -authBackend http://localhost:8080 -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr “” -prometheusListenAddr localhost:9229 -secretPath /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret -logFormat json -config config.toml
├─2409 /opt/gitlab/embedded/bin/gitaly-wrapper /opt/gitlab/embedded/bin/gitaly serve /var/opt/gitlab/gitaly/config.toml
├─2410 /opt/gitlab/embedded/bin/postgres_exporter --no-collector.stat_user_tables --collector.postmaster --web.listen-address=localhost:9187 --extend.query-path=/var/opt/gitlab/postgres-exporter/queries.yaml
├─2411 /opt/gitlab/embedded/bin/registry serve ./config.yml
├─2415 svlogd -tt /var/log/gitlab/gitlab-exporter
├─2416 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/gitlab-exporter web -c /var/opt/gitlab/gitlab-exporter/gitlab-exporter.yml
├─2417 svlogd -tt /var/log/gitlab/redis
├─2418 “/opt/gitlab/embedded/bin/redis-server unixsocket:/var/opt/gitlab/redis/redis.socket”
├─2419 svlogd -tt /var/log/gitlab/logrotate
├─2420 svlogd -tt /var/log/gitlab/postgresql
├─2421 svlogd -tt /var/log/gitlab/gitlab-kas
├─2422 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
├─2423 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
├─2424 /opt/gitlab/embedded/bin/gitlab-kas --configuration-file /var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml
├─2426 svlogd -tt /var/log/gitlab/redis-exporter
├─2427 svlogd -tt /var/log/gitlab/prometheus
├─2430 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///var/opt/gitlab/redis/redis.socket
├─2431 /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitlab/prometheus/prometheus.yml
├─2441 sleep 600
├─2512 “nginx: worker process”
├─2513 “nginx: worker process”
├─2514 “nginx: worker process”
├─2515 “nginx: worker process”
├─2516 “nginx: worker process”
├─2517 “nginx: worker process”
├─2518 “nginx: worker process”
├─2519 “nginx: worker process”
├─2520 “nginx: cache manager process”
├─2534 /opt/gitlab/embedded/bin/gitaly serve /var/opt/gitlab/gitaly/config.toml
├─2546 "postgres: checkpointer "
├─2547 "postgres: background writer "
├─2548 "postgres: walwriter "
├─2549 "postgres: autovacuum launcher "
├─2550 "postgres: stats collector "
├─2551 "postgres: logical replication launcher "
└─2767 “postgres: gitlab-psql gitlabhq_production [local] idle”

Dec 03 10:46:46 [domain] systemd[1]: Started gitlab-runsvdir.service - GitLab Runit supervision process.

Helpful resources

Tried all

Gitlab not working after reboot
GitLab VM Unresponsive (#33854) · Issues · GitLab.org / GitLab FOSS · GitLab
Troubleshooting Gitlab Crashes - #5 by iwalker
Gitlab is not starting up - technical break - #5 by MichDem
GITLAB not coming up after VM restart - #4 by grove
Works after installation but gets 502 error after first restart - #4 by grove
postgresql - Gitlab not starting after upgrade to Ubuntu 18.04 - Stack Overflow
Gitlab-runsvdir service failed to start with Ubuntu 18.04 GUI edition - #2 by dnsmichi
GitLab not starting after upgrading from Ubuntu 14.04 - #7 by iwalker
After a period of inactivity gitlab becomes unresponsive to the point of locking the server completely (#21915) · Issues · GitLab.org / GitLab · GitLab
Troubleshooting GitLab Runner | GitLab
Troubleshooting Omnibus GitLab installation issues | GitLab
Triggering specific job causes freeze of gitlab machine
How to restart GitLab | GitLab