After upgrading from 16.6.2 to 16.7.2, puma worker process consumes 100% CPU

When I tried to upgrade from 16.6.2 to 16.7.2, Gitlab UI was down, getting 502. Ran top and noticed Puma worker processes consuming 100% CPU.
Below is the error logs for /var/log/gitlab/puma

=== puma startup: 2024-01-14 13:25:03 +0100 ===
bundler: failed to load command: puma (/opt/gitlab/embedded/bin/puma)
/ssd_storage/gitlab/opt/embedded/lib/ruby/gems/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:72:in `_fork': Cannot allocate memory - fork(2) (Errno::ENOMEM)
        from /ssd_storage/gitlab/opt/embedded/lib/ruby/gems/3.1.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:72:in `_fork'
        from /ssd_storage/gitlab/opt/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8/lib/active_support/fork_tracker.rb:7:in `_fork'
        from /ssd_storage/gitlab/opt/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/cluster.rb:99:in `fork'
        from /ssd_storage/gitlab/opt/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/cluster.rb:99:in `spawn_worker'
        from /ssd_storage/gitlab/opt/embedded/lib/ruby/gems/3.1.0/gems/puma-6.4.0/lib/puma/cluster.rb:81:in `block in spawn_workers'

After checking the gitlab.rb config, I set the puma worker processes to 3 from 8(default i guess) and gitlab UI comes up again. But that is a temporary fix, not sure what needs to be fix manually.
I checked the memory as well and it seems fine.

My system is:

NAME=“Ubuntu”
VERSION=“18.04.6 LTS (Bionic Beaver)”
ID=ubuntu
PRETTY_NAME=“Ubuntu 18.04.6 LTS”
VERSION_ID=“18.04”
HOME_URL=“https://www.ubuntu.com/”
SUPPORT_URL=“https://help.ubuntu.com/”
BUG_REPORT_URL=“Bugs : Ubuntu”
VERSION_CODENAME=bionic

Thanks in advance for the help

fork': Cannot allocate memory - fork(2) (Errno::ENOMEM) sounds like child forks are never free’d thus exhausting memory.

Maybe related to Puma 6.4.1 causing increased errors (#437948) · Issues · GitLab.org / GitLab · GitLab which links to an upstream discussion in Puma cluster not reaping child processes when PID is 1 with Puma 6.4.1 · Issue #3313 · puma/puma · GitHub which itself links to Ruby bugs with fork/waitpid and not reaping child processes.Bug #19837: Concurrent calls to Process.waitpid2 misbehave on Ruby 3.1 & 3.2 - Ruby master - Ruby Issue Tracking System

I think that our engineers should have a look. Can you report a bug report, please? Feel free to tag me there, too.

same here. From 16.6.6 to 16.7.4