Hi, I’m stuck with my server refusing to work.
Showing this behaviour since server reboot:
- Puma restarts every 55-60 seconds.
- During that time, CPU is around 99% and memory (RES) increases monotonically from <100 MB to >700 MB.
- If I keep pressing F5, during 3-4 seconds near the restart, GitLab actually works!
status doesn’t report other anomalies:
~# gitlab-ctl status
run: crond: (pid 1088) 1091s; run: log: (pid 1075) 1091s
run: gitaly: (pid 1087) 1091s; run: log: (pid 1080) 1091s
run: gitlab-workhorse: (pid 1095) 1091s; run: log: (pid 1072) 1091s
run: logrotate: (pid 1083) 1091s; run: log: (pid 1073) 1091s
run: nginx: (pid 1089) 1091s; run: log: (pid 1076) 1091s
run: postgresql: (pid 1081) 1091s; run: log: (pid 1077) 1091s
run: puma: (pid 29364) 36s; run: log: (pid 1071) 1091s
run: redis: (pid 1085) 1091s; run: log: (pid 1078) 1091s
run: registry: (pid 1084) 1091s; run: log: (pid 1074) 1091s
run: sidekiq: (pid 1086) 1091s; run: log: (pid 1079) 1091s
Configuration is mostly default, with these changes to run in a constrained virtual machine:
puma['worker_processes'] = 0
sidekiq['max_concurrency'] = 10
postgresql['shared_buffers'] = "256MB"
prometheus_monitoring['enable'] = false
I’ve discarded (I think) interactions with process killers trying also this configuration settings:
puma['worker_timeout'] = 300
puma['worker_processes'] = 2
puma['min_threads'] = 1
puma['max_threads'] = 2
… and also assigning 4GB RAM to the vm.
Anyway, default configuration doesn’t work either.
Puma logs look like this:
==> /var/log/gitlab/puma/puma_stdout.log <==
=== puma startup: 2021-12-21 09:45:53 +0100 ===
==> /var/log/gitlab/puma/current <==
2021-12-21_08:45:53.14461 {"timestamp":"2021-12-21T08:45:53.144Z","pid":4076,"message":"Use Ctrl-C to stop"}
==> /var/log/gitlab/puma/puma_stderr.log <==
=== puma startup: 2021-12-21 09:45:53 +0100 ===
==> /var/log/gitlab/puma/current <==
2021-12-21_08:46:05.06572 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"Puma starting in single mode..."}
2021-12-21_08:46:05.06581 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"* Puma version: 5.3.2 (ruby 2.7.4-p191) (\"Sweetnighter\")"}
2021-12-21_08:46:05.06585 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"* Min threads: 1"}
2021-12-21_08:46:05.06590 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"* Max threads: 2"}
2021-12-21_08:46:05.06595 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"* Environment: production"}
2021-12-21_08:46:05.06601 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"* PID: 6325"}
2021-12-21_08:46:25.52787 WARNING: Puma is running in Single mode (workers = 0). Some features may not work. Please refer to https://gitlab.com/groups/gitlab-org/-/epics/5303 for info.
2021-12-21_08:46:52.57723 {"timestamp":"2021-12-21T08:46:52.577Z","pid":6325,"message":"* Listening on unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket"}
2021-12-21_08:46:52.57750 {"timestamp":"2021-12-21T08:46:52.577Z","pid":6325,"message":"* Listening on http://127.0.0.1:8080"}
2021-12-21_08:46:52.58077 {"timestamp":"2021-12-21T08:46:52.580Z","pid":6325,"message":"Use Ctrl-C to stop"}
==> /var/log/gitlab/puma/puma_stdout.log <==
=== puma startup: 2021-12-21 09:46:52 +0100 ===
==> /var/log/gitlab/puma/puma_stderr.log <==
=== puma startup: 2021-12-21 09:46:52 +0100 ===
I’m using GitLab CE 14.5.0.
Please, help!