I run latest GitLab-CE (gitlab-ce-12.0.3-ce.0.el7.x86_64) on CentOS 7. It is omnibus install and it ran well until yesterday. After update of some OS packages* and system restart, GitLab doesn’t work.
* List of updated packages:
* 7:device-mapper-1.02.149-10.el7_6.8.x86_64
* 7:device-mapper-event-1.02.149-10.el7_6.8.x86_64
* 7:device-mapper-event-libs-1.02.149-10.el7_6.8.x86_64
* 7:device-mapper-libs-1.02.149-10.el7_6.8.x86_64
* 7:lvm2-2.02.180-10.el7_6.8.x86_64
* 7:lvm2-libs-2.02.180-10.el7_6.8.x86_64
* gitlab-runner-12.0.2-1.x86_64
* glibc-2.17-260.el7_6.6.x86_64
* glibc-common-2.17-260.el7_6.6.x86_64
* glibc-devel-2.17-260.el7_6.6.x86_64
* glibc-headers-2.17-260.el7_6.6.x86_64
* libteam-1.27-6.el7_6.1.x86_64
* systemd-219-62.el7_6.7.x86_64
* systemd-libs-219-62.el7_6.7.x86_64
* systemd-sysv-219-62.el7_6.7.x86_64
* teamd-1.27-6.el7_6.1.x86_64
* tzdata-2019b-1.el7.noarch
Symptoms:
502 Whoops, GitLab is taking too much time to respond. (when accessing web interface)
GitLab: Failed to authorize your Git request: internal API unreachable (when trying git pull)
gitlab-ctl reconfigure and gitlab-ctl restart did not help.
In /var/log/gitlab/unicorn/unicorn_stderr.log I found:
I, [2019-07-17T10:23:20.831800 #23128] INFO – : Refreshing Gem list
I, [2019-07-17T10:24:04.558224 #23128] INFO – : listening on addr=127.0.0.1:8080 fd=19
I, [2019-07-17T10:24:04.558414 #23128] INFO – : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I, [2019-07-17T10:24:04.558640 #23128] INFO – : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=20
E, [2019-07-17T10:24:04.559419 #23128] ERROR – : Operation not permitted (Errno::EPERM)
/var/opt/gitlab/gitlab-rails/etc/unicorn.rb:47:inkill' /var/opt/gitlab/gitlab-rails/etc/unicorn.rb:47:inblock in reload’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:543:inspawn_missing_workers' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:instart’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in<top (required)>' /opt/gitlab/embedded/bin/unicorn:23:inload’
/opt/gitlab/embedded/bin/unicorn:23:in<top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:inload’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:inkernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:inrun’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:inexec' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:inrun’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:ininvoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:indispatch’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:indispatch' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:instart’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:instart' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:inblock in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:inwith_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in<top (required)>’
/opt/gitlab/embedded/bin/bundle:23:inload' /opt/gitlab/embedded/bin/bundle:23:in’
What is wrong with my unicorn and how to repair it?
I found that rubby is runnig as git user:
$ sudo ps -ef | grep ruby
git 22951 22928 0 10:23 ? 00:00:06 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 22928 /tmp/gitaly-ruby531460471/socket.0
git 22952 22928 0 10:23 ? 00:00:06 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 22928 /tmp/gitaly-ruby531460471/socket.1
and sockets are owned by git user too:
$ ls -l /tmp/gitaly-ruby531460471/socket.{0,1} /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
srwxr-xr-x. 1 git git 0 Jul 17 10:23 /tmp/gitaly-ruby531460471/socket.0
srwxr-xr-x. 1 git git 0 Jul 17 10:23 /tmp/gitaly-ruby531460471/socket.1
srwxrwxrwx. 1 git git 0 Jul 17 11:32 /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
Port 8080 is not in use:
$ sudo ss -t -u -n | grep -e Netid -e 8080
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
$
gitlab-ctl status tells nothing special:
$ sudo gitlab-ctl status
run: alertmanager: (pid 22911) 9025s; run: log: (pid 4176) 14105s
run: gitaly: (pid 22922) 9025s; run: log: (pid 4212) 14105s
run: gitlab-monitor: (pid 22941) 9024s; run: log: (pid 4205) 14105s
run: gitlab-workhorse: (pid 22943) 9024s; run: log: (pid 4220) 14105s
run: grafana: (pid 22960) 9023s; run: log: (pid 4232) 14105s
run: logrotate: (pid 2143) 1823s; run: log: (pid 4225) 14105s
run: nginx: (pid 22982) 9023s; run: log: (pid 4208) 14105s
run: node-exporter: (pid 22988) 9022s; run: log: (pid 4222) 14105s
run: postgres-exporter: (pid 22994) 9022s; run: log: (pid 4157) 14105s
run: postgresql: (pid 23000) 9021s; run: log: (pid 4156) 14105s
run: prometheus: (pid 23003) 9021s; run: log: (pid 4227) 14105s
run: redis: (pid 23096) 9020s; run: log: (pid 4155) 14105s
run: redis-exporter: (pid 23100) 9020s; run: log: (pid 4159) 14105s
run: sidekiq: (pid 23107) 9020s; run: log: (pid 4230) 14105s
run: unicorn: (pid 5112) 44s; run: log: (pid 4152) 14105s