Gitlab 502 error; puma restarting every minute

How to Use GitLab Self-managed
1

Hi, I’m stuck with my server refusing to work.

Showing this behaviour since server reboot:

  • Puma restarts every 55-60 seconds.
  • During that time, CPU is around 99% and memory (RES) increases monotonically from <100 MB to >700 MB.
  • If I keep pressing F5, during 3-4 seconds near the restart, GitLab actually works!

status doesn’t report other anomalies:

~# gitlab-ctl status
run: crond: (pid 1088) 1091s; run: log: (pid 1075) 1091s
run: gitaly: (pid 1087) 1091s; run: log: (pid 1080) 1091s
run: gitlab-workhorse: (pid 1095) 1091s; run: log: (pid 1072) 1091s
run: logrotate: (pid 1083) 1091s; run: log: (pid 1073) 1091s
run: nginx: (pid 1089) 1091s; run: log: (pid 1076) 1091s
run: postgresql: (pid 1081) 1091s; run: log: (pid 1077) 1091s
run: puma: (pid 29364) 36s; run: log: (pid 1071) 1091s
run: redis: (pid 1085) 1091s; run: log: (pid 1078) 1091s
run: registry: (pid 1084) 1091s; run: log: (pid 1074) 1091s
run: sidekiq: (pid 1086) 1091s; run: log: (pid 1079) 1091s

Configuration is mostly default, with these changes to run in a constrained virtual machine:

puma['worker_processes'] = 0
sidekiq['max_concurrency'] = 10
postgresql['shared_buffers'] = "256MB"
prometheus_monitoring['enable'] = false

I’ve discarded (I think) interactions with process killers trying also this configuration settings:

puma['worker_timeout'] = 300
puma['worker_processes'] = 2
puma['min_threads'] = 1
puma['max_threads'] = 2

… and also assigning 4GB RAM to the vm.

Anyway, default configuration doesn’t work either.

Puma logs look like this:

==> /var/log/gitlab/puma/puma_stdout.log <==
=== puma startup: 2021-12-21 09:45:53 +0100 ===

==> /var/log/gitlab/puma/current <==
2021-12-21_08:45:53.14461 {"timestamp":"2021-12-21T08:45:53.144Z","pid":4076,"message":"Use Ctrl-C to stop"}

==> /var/log/gitlab/puma/puma_stderr.log <==
=== puma startup: 2021-12-21 09:45:53 +0100 ===

==> /var/log/gitlab/puma/current <==
2021-12-21_08:46:05.06572 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"Puma starting in single mode..."}
2021-12-21_08:46:05.06581 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"* Puma version: 5.3.2 (ruby 2.7.4-p191) (\"Sweetnighter\")"}
2021-12-21_08:46:05.06585 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"*  Min threads: 1"}
2021-12-21_08:46:05.06590 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"*  Max threads: 2"}
2021-12-21_08:46:05.06595 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"*  Environment: production"}
2021-12-21_08:46:05.06601 {"timestamp":"2021-12-21T08:46:05.065Z","pid":6325,"message":"*          PID: 6325"}
2021-12-21_08:46:25.52787 WARNING: Puma is running in Single mode (workers = 0). Some features may not work. Please refer to https://gitlab.com/groups/gitlab-org/-/epics/5303 for info.
2021-12-21_08:46:52.57723 {"timestamp":"2021-12-21T08:46:52.577Z","pid":6325,"message":"* Listening on unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket"}
2021-12-21_08:46:52.57750 {"timestamp":"2021-12-21T08:46:52.577Z","pid":6325,"message":"* Listening on http://127.0.0.1:8080"}
2021-12-21_08:46:52.58077 {"timestamp":"2021-12-21T08:46:52.580Z","pid":6325,"message":"Use Ctrl-C to stop"}

==> /var/log/gitlab/puma/puma_stdout.log <==
=== puma startup: 2021-12-21 09:46:52 +0100 ===

==> /var/log/gitlab/puma/puma_stderr.log <==
=== puma startup: 2021-12-21 09:46:52 +0100 ===

I’m using GitLab CE 14.5.0.

Please, help!

4

As an aside, check your Gitlab instance isn’t affected or compromised by the CVE’s that have been posted recently on this forum. Because it’s not normal for puma or high cpu usage normally.

Alternatively, make sure you have 4cpu, and 8gb ram for the specs of the machine, anything less generally causes problems.

I’ve asked for Gitlab admins/mods to remove the spam flag, of which they have kindly done.

5

Did you find a solution? I have frequent 502’s as well.