I updated GitLab from version 11.10.8 to 13.0.0 by following a recommended upgrade path given at https://docs.gitlab.com/ee/policy/maintenance.html#patch-releases. The system runs Ubuntu 20.04 and the update was done using apt commands of form “apt install gitlab-ee=xx.yy.z-ee.0”, where the version xx.yy.z went through the chain 11.10.8 -> 11.11.8 -> 12.0.12 -> 12.10.6 -> 13.0.0 during the upgrade path.
Now the GitLab repositories still seem to work from the command line, but I can no longer access the GitLab web interface. The result is just 500 with the message “Whoops, something went wrong on our end.” gitlab-ctl status gives:
run: alertmanager: (pid 218337) 1456s; run: log: (pid 187061) 3946s
run: gitaly: (pid 218355) 1455s; run: log: (pid 187072) 3946s
run: gitlab-exporter: (pid 218383) 1455s; run: log: (pid 187060) 3946s
down: gitlab-monitor: 0s, normally up, want up; run: log: (pid 187066) 3946s
run: gitlab-workhorse: (pid 218535) 1424s; run: log: (pid 187062) 3946s
run: grafana: (pid 218548) 1423s; run: log: (pid 187067) 3946s
run: logrotate: (pid 218564) 1423s; run: log: (pid 187064) 3946s
run: nginx: (pid 218570) 1423s; run: log: (pid 187069) 3946s
run: node-exporter: (pid 218583) 1422s; run: log: (pid 187059) 3946s
run: postgres-exporter: (pid 218589) 1422s; run: log: (pid 187065) 3946s
run: postgresql: (pid 218596) 1421s; run: log: (pid 187070) 3946s
run: prometheus: (pid 218599) 1421s; run: log: (pid 187068) 3946s
run: redis: (pid 218615) 1421s; run: log: (pid 187058) 3946s
run: redis-exporter: (pid 218630) 1420s; run: log: (pid 187071) 3946s
run: sidekiq: (pid 218637) 1420s; run: log: (pid 187063) 3946s
run: unicorn: (pid 218644) 1419s; run: log: (pid 187073) 3946s
So everything else is running except gitlab-monitor. gitlab-ctl tail contains the following reason:
2020-06-04_20:15:57.12026 chpst: fatal: unable to run: /opt/gitlab/embedded/bin/gitlab-mon: file does not exist
It seems that 13.0.0 no longer has gitlab-monitor: it has been renamed to gitlab-exporter: https://dev.gitlab.org/gitlab-org/gitlab-exporter/commit/c071e828c275db41fe77b5338c31bffd84b0a178
So why would gitlab-ctl status still show an entry for it? Trying gitlab-ctl reconfigure ends in an error that provides zero useful information about what actually goes wrong:
There was an error running gitlab-ctl reconfigure:
undefined method `’ for nil:NilClass
How could I try to resolve this issue? I e.g. cannot downgrade GitLab back to a lower version: the attempt fails with en error about incompatible postgresql version. Trying to upgrade fails due to having a corrupt configuration (the upgrade tells to first run gitlab-ctl reconfigure).
UPDATE: I now did also “gitlab-rake gitlab:env:info”:
System information
System: Ubuntu 20.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.6.6p146
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.9
Git Version: 2.26.2
Sidekiq Version:5.2.7
Go Version: unknown
GitLab information
Version: 13.0.5-ee
Revision: 260c6231ed0
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.7
URL: https://a.b.c.d
HTTP Clone URL: https://a.b.c.d/some-group/some-project.git
SSH Clone URL: git@a.b.c.d:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.2.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Above the real IP-addresses were replaced by “a.b.c.d”.