GitLab unicorn doesn't listen on 8080 (after yum update)

How to Use GitLab
1

I run latest GitLab-CE (gitlab-ce-12.0.3-ce.0.el7.x86_64) on CentOS 7. It is omnibus install and it ran well until yesterday. After update of some OS packages* and system restart, GitLab doesn’t work.

* List of updated packages:

* 7:device-mapper-1.02.149-10.el7_6.8.x86_64
* 7:device-mapper-event-1.02.149-10.el7_6.8.x86_64
* 7:device-mapper-event-libs-1.02.149-10.el7_6.8.x86_64
* 7:device-mapper-libs-1.02.149-10.el7_6.8.x86_64
* 7:lvm2-2.02.180-10.el7_6.8.x86_64
* 7:lvm2-libs-2.02.180-10.el7_6.8.x86_64
* gitlab-runner-12.0.2-1.x86_64
* glibc-2.17-260.el7_6.6.x86_64
* glibc-common-2.17-260.el7_6.6.x86_64
* glibc-devel-2.17-260.el7_6.6.x86_64
* glibc-headers-2.17-260.el7_6.6.x86_64
* libteam-1.27-6.el7_6.1.x86_64
* systemd-219-62.el7_6.7.x86_64
* systemd-libs-219-62.el7_6.7.x86_64
* systemd-sysv-219-62.el7_6.7.x86_64
* teamd-1.27-6.el7_6.1.x86_64
* tzdata-2019b-1.el7.noarch

Symptoms:

502 Whoops, GitLab is taking too much time to respond. (when accessing web interface)

GitLab: Failed to authorize your Git request: internal API unreachable (when trying git pull)

gitlab-ctl reconfigure and gitlab-ctl restart did not help.

In /var/log/gitlab/unicorn/unicorn_stderr.log I found:

I, [2019-07-17T10:23:20.831800 #23128] INFO – : Refreshing Gem list
I, [2019-07-17T10:24:04.558224 #23128] INFO – : listening on addr=127.0.0.1:8080 fd=19
I, [2019-07-17T10:24:04.558414 #23128] INFO – : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I, [2019-07-17T10:24:04.558640 #23128] INFO – : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=20
E, [2019-07-17T10:24:04.559419 #23128] ERROR – : Operation not permitted (Errno::EPERM)
/var/opt/gitlab/gitlab-rails/etc/unicorn.rb:47:in kill' /var/opt/gitlab/gitlab-rails/etc/unicorn.rb:47:in block in reload’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:543:in spawn_missing_workers' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in start’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in <top (required)>' /opt/gitlab/embedded/bin/unicorn:23:in load’
/opt/gitlab/embedded/bin/unicorn:23:in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in load’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in kernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in run’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in exec' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in start' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in block in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

What is wrong with my unicorn and how to repair it?

I found that rubby is runnig as git user:

$ sudo ps -ef | grep ruby
git 22951 22928 0 10:23 ? 00:00:06 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 22928 /tmp/gitaly-ruby531460471/socket.0
git 22952 22928 0 10:23 ? 00:00:06 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 22928 /tmp/gitaly-ruby531460471/socket.1

and sockets are owned by git user too:

$ ls -l /tmp/gitaly-ruby531460471/socket.{0,1} /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
srwxr-xr-x. 1 git git 0 Jul 17 10:23 /tmp/gitaly-ruby531460471/socket.0
srwxr-xr-x. 1 git git 0 Jul 17 10:23 /tmp/gitaly-ruby531460471/socket.1
srwxrwxrwx. 1 git git 0 Jul 17 11:32 /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket

Port 8080 is not in use:

$ sudo ss -t -u -n | grep -e Netid -e 8080
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
$

gitlab-ctl status tells nothing special:

$ sudo gitlab-ctl status
run: alertmanager: (pid 22911) 9025s; run: log: (pid 4176) 14105s
run: gitaly: (pid 22922) 9025s; run: log: (pid 4212) 14105s
run: gitlab-monitor: (pid 22941) 9024s; run: log: (pid 4205) 14105s
run: gitlab-workhorse: (pid 22943) 9024s; run: log: (pid 4220) 14105s
run: grafana: (pid 22960) 9023s; run: log: (pid 4232) 14105s
run: logrotate: (pid 2143) 1823s; run: log: (pid 4225) 14105s
run: nginx: (pid 22982) 9023s; run: log: (pid 4208) 14105s
run: node-exporter: (pid 22988) 9022s; run: log: (pid 4222) 14105s
run: postgres-exporter: (pid 22994) 9022s; run: log: (pid 4157) 14105s
run: postgresql: (pid 23000) 9021s; run: log: (pid 4156) 14105s
run: prometheus: (pid 23003) 9021s; run: log: (pid 4227) 14105s
run: redis: (pid 23096) 9020s; run: log: (pid 4155) 14105s
run: redis-exporter: (pid 23100) 9020s; run: log: (pid 4159) 14105s
run: sidekiq: (pid 23107) 9020s; run: log: (pid 4230) 14105s
run: unicorn: (pid 5112) 44s; run: log: (pid 4152) 14105s

2

I got a working Gitlab by doing almost complete uninstall and reinstallation of the code and restoring data from backup.