Unable to login (error code 500) after Ubuntu upgrade from 18.04 to 20.04

We had a server running Ubuntu 18.04 and Gitlab Omnibus 9.4.5, where Gitlab was working just fine.

I upgraded the server to Ubuntu 20.04, but made no changes to Gitlab, it was not updated as part of the process.

Now I am not able to login via the web login-page anymore…

sudo gitlab-rake gitlab:env:info:

System information
System: Ubuntu 20.04
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.13.4
Sidekiq Version:5.0.0
Go Version: unknown

GitLab information
Version: 9.4.5
Revision: 140292e
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://gitlab
HTTP Clone URL: http://gitlab/some-group/some-project.git
SSH Clone URL: git@gitlab:some-group/some-project.git
Using LDAP: no
Using Omniauth: no

GitLab Shell
Version: 5.3.1
Repository storage paths:

  • default: /var/opt/gitlab/git-data/repositories
    Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
    Git: /opt/gitlab/embedded/bin/git

When I try to log in I see the following in “gitlab-ctl tail”:

==> /var/log/gitlab/nginx/gitlab_access.log <== - - [11/Nov/2020:15:07:56 +0100] “GET /users/sign_in HTTP/1.1” 200 3918 “-” “Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0”

==> /var/log/gitlab/gitlab-monitor/current <==
2020-11-11_14:07:58.75454 ::1 - - [11/Nov/2020:15:07:58 CET] “GET /sidekiq HTTP/1.1” 200 3399
2020-11-11_14:07:58.75476 - -> /sidekiq
2020-11-11_14:08:00.68032 ::1 - - [11/Nov/2020:15:08:00 CET] “GET /database HTTP/1.1” 200 42039
2020-11-11_14:08:00.68042 - -> /database

==> /var/log/gitlab/node-exporter/current <==
2020-11-11_14:08:02.13219 time=“2020-11-11T15:08:02+01:00” level=error msg=“ERROR: diskstats collector failed after 0.000567s: invalid line for /proc/diskstats for xvda” source=“node_exporter.go:95”

==> /var/log/gitlab/gitlab-monitor/current <==
2020-11-11_14:08:03.09177 ::1 - - [11/Nov/2020:15:08:02 CET] “GET /process HTTP/1.1” 200 1138
2020-11-11_14:08:03.09189 - -> /process

==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST “/users/sign_in” for at 2020-11-11 15:08:05 +0100
Processing by SessionsController#create as HTML
Parameters: {“utf8”=>“✓”, “authenticity_token”=>“fKvm7FPkcMkJ+b2rE0CW1Vor0mdjPX5Ge/48UGytxMhukSvxbzTyG9h2F+hoPuTZMIi3oOrNXKB/OpxBaZhFPA==”, “user”=>{“login”=>“jarle”, “password”=>"[FILTERED]", “remember_me”=>“0”}}
Completed 401 Unauthorized in 268ms (ActiveRecord: 73.1ms)
Processing by SessionsController#new as HTML
Parameters: {“utf8”=>“✓”, “authenticity_token”=>“fKvm7FPkcMkJ+b2rE0CW1Vor0mdjPX5Ge/48UGytxMhukSvxbzTyG9h2F+hoPuTZMIi3oOrNXKB/OpxBaZhFPA==”, “user”=>{“login”=>“jarle”, “password”=>"[FILTERED]", “remember_me”=>“0”}}
Completed 500 Internal Server Error in 71ms (ActiveRecord: 0.8ms)

BCrypt::Errors::InvalidHash (invalid hash):
app/controllers/sessions_controller.rb:33:in new' lib/gitlab/i18n.rb:45:in with_locale’
lib/gitlab/i18n.rb:51:in with_user_locale' app/controllers/application_controller.rb:294:in set_locale’
lib/gitlab/request_context.rb:18:in call' lib/gitlab/metrics/requests_rack_middleware.rb:27:in call’

Is there a problem with the password hash somehow?

I know this an old version of Gitlab, but I would like to fix this authentication problem before I even start trying to upgrade Gitlab itself…

How do I debug/fix this problem?

Is there a problem with the password hash somehow?

That is the affected area, but the issue is likely not your data or states. You’re instead running into an issue caused due to old bundled dependencies in your GitLab installation now running against upgraded system libraries that have introduced certain backward incompatibilities.

How do I debug/fix this problem?

You can try to apply the one-liner hack-change suggested here to return the breaking function back to its older behavior, then perform an upgrade to resolve it cleanly. The path to the file being edited there will be under your omnibus installation path - just look for the same filename/path structure of the dependency and then attempt an edit after making a backup first.

1 Like

Thank you for your help.

I just started upgrading Gitlab, without applying any hacks, and when I reached v 11.11.8 the problem was fixed and it started working again.

1 Like