I tried to upgrade from 16.6.6 to 16.7.4, but puma worker can’t start with “cannot allocate memory”. There is only one workaround, when I switch to single node with puma[‘worker_process’] = 0
puma related settings:
puma[‘port’] = 8085
puma[‘worker_processes’] = 0
puma[‘per_worker_max_memory_mb’] = 512
CentOS 7, omnibus package, 8GB RAM.
Thanks for your help!
Have you checked the memory usage on your CentOS 7 server using the free -m command? You should ensure that there is sufficient free memory available to run GitLab and its components. It did the trick for me.
Hi @Hutius I have enough free memory I guess:
This is a dev server with minimal load, without similar problem in the past. Why does it happen at 16.7 upgrade?
I’ve never had this, but then I am using the default puma settings. I would try perhaps setting:
puma['per_worker_max_memory_mb'] = 1024
as maybe 512 is a little too low. Perhaps then increases the worker_processes from zero at this point to the number you were using before.
8GB of ram is fine, my instance uses about 4GB of ram by default, so you may then wish to take a look at how many puma processes you are attempting to run, and if you have enough available ram to do that. If we assume 4 worker_processes at 1024 then we might be reaching the limit. Or if 8 worker processes and 512. It could be that in conjunction with all the other Gitlab services, that there isn’t enough ram available.
What were your worker processes set to before you reset it to zero? This would help confirm if what I think is the case.
puma[‘per_worker_max_memory_mb’] = 512 - I tried this only after this issue.
Originally I only used the port related puma setting, when this issue was appeared.
So the worker processes was default value 8, and max memory was 1200 mb.
With default settings, I have never had such problem.
You have 8cpu/vcpu in your server? I’ve got 4cpu, and it’s showing 4 worker processes with the default of 1024 for the per_worker_max_memory. The docs suggest: Configure the bundled Puma instance of the GitLab package | GitLab
Therefore, if you do have 8cpu in your server, that would suggest that the puma per_worker_max_memory changed somewhere in memory requirements at a certain Gitlab version to 1024 and potentially caused the problem.
If you have configured 8 worker_processes in the config file, this isn’t the default - the default values in /opt/gitlab/etc/gitlab.rb.template suggest:
# puma['worker_processes'] = 2
###! Docs: https://github.com/schneems/puma_worker_killer
# puma['per_worker_max_memory_mb'] = 1024
The worker max memory fits with the Gitlab docs I linked.
I’m expecting the commented worker_processes here potentially was a default at some point, however it seems that depending on how many cpu in the server, this value can increase on it’s own without having to be configured manually.
If your server doesn’t have 8cpu, then this has most likely been configured incorrectly, and it the value is commented, then it will run worker processes based on the count if your cpu. An example from mine when I checked:
root@gitlab:~# ps aux | grep puma | grep worker
git 195191 0.0 10.5 1076396 855980 ? Ssl Feb05 2:07 puma 6.4.0 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,tcp://127.0.0.1:8080) [gitlab-puma-worker]
git 195296 0.0 10.4 1210692 848692 ? Sl Feb05 1:41 puma: cluster worker 0: 195191 [gitlab-puma-worker]
git 195299 0.0 10.4 1196352 845804 ? Sl Feb05 1:30 puma: cluster worker 1: 195191 [gitlab-puma-worker]
git 195311 0.0 10.5 1215812 854560 ? Sl Feb05 1:38 puma: cluster worker 2: 195191 [gitlab-puma-worker]
git 195323 0.0 10.4 1088724 847532 ? Sl Feb05 0:34 puma: cluster worker 3: 195191 [gitlab-puma-worker]
the first process just being the socket file, the remaining being the 4 worker processes, which fit’s since I have 4cpu. And from my config:
root@gitlab:~# cat /etc/gitlab/gitlab.rb | grep puma | grep worker_proc
#puma['worker_processes'] = 0
this is commented out, but the value here is zero simply because I was experimenting with a low resource setup for running on something like a Raspberry PI.
Indeed, it looks like you have enough memory for that. Well, then, I suggest instead of jumping to a higher number of worker processes, gradually increase the number from 0 to find the optimal number that your system can support without running into memory issues. Start with puma[‘worker_processes’] = 1 and monitor the memory usage.
Hi @iwalker you’re right I have 4 vcpu, and puma wanted to start with 4 worker, but it couldn’t if I comment out the puma settings.
puma[‘port’] = 8085
#puma[‘worker_processes’] = 0
#puma[‘per_worker_max_memory_mb’] = 1024
But the fact is that, with the default settings, puma can’t start, restarting continously:
2024-02-08_12:35:35.51607 {"timestamp":"2024-02-08T12:35:35.515Z","pid":6818,"message":"* Listening on unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket"}
2024-02-08_12:35:35.51647 {"timestamp":"2024-02-08T12:35:35.516Z","pid":6818,"message":"* Listening on http://127.0.0.1:8085"}
2024-02-08_12:35:35.51665 {"timestamp":"2024-02-08T12:35:35.516Z","pid":6818,"message":"! WARNING: Detected 2 Thread(s) started in app boot:"}
2024-02-08_12:35:35.51680 {"timestamp":"2024-02-08T12:35:35.516Z","pid":6818,"message":"! #\u003cThread:0x00007fe1ce231538 /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:91:in `sleep'"}
2024-02-08_12:35:35.51693 {"timestamp":"2024-02-08T12:35:35.516Z","pid":6818,"message":"! #\u003cThread:0x00007fe177e78a28 /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/session_flusher.rb:81 sleep\u003e - /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/session_flusher.rb:83:in `sleep'"}
2024-02-08_12:35:35.51713 {"timestamp":"2024-02-08T12:35:35.517Z","pid":6818,"message":"Use Ctrl-C to stop"}
2024-02-08_12:35:38.45807 {"timestamp":"2024-02-08T12:35:38.448Z","pid":7003,"message":"Puma starting in cluster mode..."}
2024-02-08_12:35:38.45826 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Puma version: 6.4.0 (ruby 3.1.4-p223) (\"The Eagle of Durango\")"}
2024-02-08_12:35:38.45835 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Min threads: 4"}
2024-02-08_12:35:38.45842 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Max threads: 4"}
2024-02-08_12:35:38.45849 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Environment: production"}
2024-02-08_12:35:38.45865 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Master PID: 7003"}
2024-02-08_12:35:38.45877 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Workers: 4"}
2024-02-08_12:35:38.45888 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Restarts: (✔) hot (✖) phased"}
2024-02-08_12:35:38.45901 {"timestamp":"2024-02-08T12:35:38.458Z","pid":7003,"message":"* Preloading application"}
bundler: failed to load command: puma (/opt/gitlab/embedded/bin/puma)
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:72:in `_fork': Cannot allocate memory - fork(2) (Errno::ENOMEM)
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:72:in `_fork'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/fork_tracker.rb:7:in `_fork'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/cluster.rb:99:in `fork'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/cluster.rb:99:in `spawn_worker'
Only started when I set 0 worker.
I google that error:
Cannot allocate memory - fork(2) (Errno::ENOMEM)
found this stackoverflow about ruby:
Can you try increasing the memory in your server to say maybe 16GB instead of 8GB? It seems something wants more ram that it cannot get. The other alternative is to stick with the config you have that is working, and then upgrade to 16.8.x and then try again just in case that issue was addressed in later Gitlab versions.