Upgrading GitLab crashed it. Solution found, but still not upgrading Prometheus

Observability
, ,
1

When upgrading GitLab from version 11.4.5-ee.0 to 11.6.3-ee.0 on a clean Debian Stretch release, GitLab complains about not upgrading prometheus and breaks GitLab, so that it is not accessible by the web interface any more. It shows a “502 Server is too busy to respond”. Although starting GitLab with the command “gitlab-ctl restart” is successful and shows no direct error.

Taking a closer look, unicorn tries to restart every second (seen in “gitlab-ctl tail”), because it complains about not finding the user “git:”. The reason is, that the group name “git” is not passed correctly to the environment. Fix it with inserting the line "group=git " below the line "user=git " in the file “/opt/gitlab/service/unicorn/run”.

Furthermore can prometheus still not be upgraded from version 1.x to 2.x (see log below). The problem is described in this page, but cannot be solved that way:

Neither can “gitlab-ctl prometheus-upgrade” fix it, nor with the parameter “–skip-data-migration”. I am not even using prometheus, and also tried to enable it for successful upgrade.

So after upgrading and fixing the group, it starts again, but prometheus is still causing “gitlab-ctl reconfigure” to fail.

Hope that helps to fix the group problem, and hoping for help on the prometheus upgrade.
Greetings, mtmayr2

gitlab-ctl prometheus-upgrade
Traceback (most recent call last):
7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in <main>' 6: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in load’
5: from /data/gitlab/opt/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in <top (required)>' 4: from /data/gitlab/opt/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in run’
3: from /data/gitlab/opt/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:197:in block in add_command' 2: from /opt/gitlab/embedded/service/omnibus-ctl/prometheus-upgrade.rb:35:in block in load_file’
1: from /data/gitlab/opt/embedded/service/omnibus-ctl/lib/gitlab_ctl/prometheus_upgrade.rb:22:in is_version_2?' /data/gitlab/opt/embedded/service/omnibus-ctl/lib/gitlab_ctl/prometheus_upgrade.rb:31:in version_string_check’: undefined method `’ for nil:NilClass (NoMethodError)