500 error on some users with MFA enabled

Some of our users on our GitLab CE (Omnibus) installation can’t log in to GitLab. We force MFA and after they have provided username and password they are flashed with the 500-error page instead of the expected MFA input page.
The same 500-error page is shown if we try to Impersonate the user as a GitLab admin.

Eventually these logs can help:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/profile/two_factor_auth" for 10.0.2.207 at 2017-02-27 16:26:03 +0000
Processing by Profiles::TwoFactorAuthsController#show as HTML
Completed 500 Internal Server Error in 14ms (ActiveRecord: 1.2ms)

OpenSSL::Cipher::CipherError (bad decrypt):
 app/controllers/profiles/two_factor_auths_controller.rb:5:in `show'
 lib/gitlab/middleware/multipart.rb:93:in `call'
 lib/gitlab/request_profiler/middleware.rb:15:in `call'
 lib/gitlab/middleware/go.rb:16:in `call'
1 Like

I had a similar issue, I was getting the OpenSSL::Cipher::CipherError (bad decrypt): error for one user when they attempted to enable MFA for their user account.

Digging in through the rails console I found that this users “encrypted_otp_secret_salt” value did not look like others. like a base64 value that started with “_” and ends with “==”

I found on Rails console | GitLab a function that could be used to reset all of the OTP values to nil, which apparently is a safe thing to do.

Once I reset the users OTP values to nil with:

def disable_two_factor!
  transaction do
    update(
      otp_required_for_login:      false,
      encrypted_otp_secret:        nil,
      encrypted_otp_secret_iv:     nil,
      encrypted_otp_secret_salt:   nil,
      otp_grace_period_started_at: nil,
      otp_backup_codes:            nil
    )
    self.u2f_registrations.destroy_all # rubocop: disable DestroyAll
  end
end

user = User.find_by(username: '<UserName>')

user.disable_two_factor!

Then the user can access the MFA page and can setup MFA successfully.

Other commands that were helpful were to dump a working and nonworking users properties for comparison with:

u = User.find_by_username('<UserName>')
pp u.attributes

access the rails console with: sudo gitlab-rails console

exit with Control - D…