Puma restart frequently after upgrading from v14ee to V16.8

How to Use GitLab Self-managed
1

After upgrading from version 14 to 16.8
It is found puma restart frequently. It there any means to fix the problem.
Any setting in gitlab.rb and help?

2

Hi,

It seems I have the same issue here.
Frequently, I have Http 502 error “Waiting for gitlab to boot”.

In logs, I can see that Puma restart frequently :

{"timestamp":"2024-02-23T10:30:06.010Z","pid":12639,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 11:31:34 +0100 ===
{"timestamp":"2024-02-23T10:37:34.559Z","pid":4137,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 11:38:57 +0100 ===
{"timestamp":"2024-02-23T10:45:01.066Z","pid":4693,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 11:46:20 +0100 ===
{"timestamp":"2024-02-23T12:37:25.025Z","pid":5185,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 13:38:34 +0100 ===
{"timestamp":"2024-02-23T14:35:45.547Z","pid":7731,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 15:36:51 +0100 ===
{"timestamp":"2024-02-23T14:50:05.690Z","pid":11356,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 15:51:13 +0100 ===
{"timestamp":"2024-02-23T15:05:26.433Z","pid":12363,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 16:06:36 +0100 ===
{"timestamp":"2024-02-23T15:12:48.571Z","pid":13301,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 16:13:56 +0100 ===
{"timestamp":"2024-02-23T15:37:09.378Z","pid":13859,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 16:38:15 +0100 ===
{"timestamp":"2024-02-23T15:50:29.262Z","pid":14660,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 16:51:35 +0100 ===
{"timestamp":"2024-02-23T16:37:49.090Z","pid":15167,"message":"- Gracefully stopping, waiting for requests to finish"}
=== puma startup: 2024-02-23 17:38:55 +0100 ===

My instance is running on a constrained environment (3vCPU + 4Go RAM) and I have configured Puma to work in single mode.

Thank you.

3

We were experiencing this issue on our GitLab server, explained in this issue.

The problem was this line in our gitlab.rb file.

puma['worker_processes'] = 0

Simply removing it (reverting to the default value) solved our crashes!

1 Like
4

I also tried recently to try and run Gitlab in a memory constrained environment, and the documents recommended setting puma worker_processes to zero, which caused more problems than it actually solved. I also reverted to the defaults, simply because it wasn’t possible to use Gitlab that way. As @MattTheCuber also found out.

I actually found it better instead to configure it this way to reduce memory usage:

puma['min_threads'] = 2
puma['max_threads'] = 2
puma['per_worker_max_memory_mb'] = 512

Puma worker processes, usually run one for each of the CPU you have in the machine, so yours will run 3. I wouldn’t suggest setting this any lower than 2 if you really have to use this. Even then, Gitlab constantly will swap to disk anyway, because it uses on average anywhere between 3.2-3.8GB of ram.

Your best bet, increase your RAM to 8GB. One of my machines is using 2CPU, 8GB ram and works perfectly fine with this. I gave up trying to get it to work on 4GB - it’s not enough.

You can also configure sysctl:

vm.overcommit_memory = 1

which should stop OOM killer from killing processes, but your system will swap to disk more, and be really slow. Whether that stops puma killing the process or not, depends on whether it was actually puma doing it, or whether Linux’s OOM killer was doing it.