Gitlab-ce_13.0.4-ce.0 Failed; stuck on puma

Hi to all !

I’m currently using the “self-managed GitLab” version on Debian for several years now.

I tried yesterday to update, and the update process stuck all the night on:
ruby_block[wait for puma service socket] action run
I stoppped the process, restarted it, looked for the logs to no avail.

When I stop/start Gitlab, the only service that doesn’t start is puma, and when I try to start it manually, I get:
warning: puma: unable to open supervise/ok: file does not exist

I’m at my wit’s end (I don’t know the ruby ecosystem), and my Gitlab instance is down at this time.

Do someone have an idea ?

PS:

• current Gitlab version: 13.0.3-ce.0
• current target version: 13.0.4-ce.0
• I assume that the version I had before the update process was the 13.0.3 as it was release the may 29, the target the june 3, and I did my update yesterday (https://about.gitlab.com/releases/categories/releases/)
• I discovered that unicorn is now superseded by puma, but I did nothing on this point: I didn’t change anything in gitlab.rb lately

Having no idea, I did some fiddling, and I got Gitlab to run.

But before reading what I did, I must warn you: I didn’t correct anything, and I didn’t understand the WHYs.
So it is both a word of advice and excuse.

Good read.

First of all, I stopped Gitlab: gitlab-ctl stop
Well it said that the file didn’t exist; so I created it:

mkdir sv/puma/supervise
cd sv/puma/supervise
mkfifo control
mkfifo ok

Next when I tried to manually start puma (with gitlab-ctl start puma), I got: fail: puma: runsv not running

A little ps ax | grep runsv showed me that runsv was running, even if I had stopped Gitlab, so:

systemctl stop runit
killall runsv
systemctl start runit

I tried to start again puma, but once again it failed on runsv, so I looked in the Gitlab directory and saw that Gitlab had its own runsv, so I started it (./embedded/bin/runsvdir-start) and on another console, I tried to start puma and this time it started.
I stopped puma, stopped ./embedded/bin/runsvdir-start (with a simple Ctrl+C).

I started Gitlab: gitlab-ctl start
I waited, was able to logged in, then logged out.

Then I reconfigured Gitlab (as it previously failed at this step in the update): gitlab-ctl reconfigure

I restarted Gitlab: gitlab-ctl restart
I waited, was able to logged in, then logged out.

And finally, I was able to perform the update:
apt-get clean && apt-get update && apt-get install gitlab-ce

Hi @lemmel and welcome to the GitLab community forum!

Nice job fiddling, I’m glad you could get it working again!
Thanks for sharing all the relevant details, and updating the thread with your solution.

We weren’t able to help you this time, but please don’t hesitate to post here asking for help if you encounter any more bumps in the road.