MFA is completely broken for all users, unable to login

I am running Gitlab CE 15.10.3.
I am experienced with Gitlab, been running it for about 12 years now.

We all use MFA for login. Now, everyone’s MFA is broken. Gitlab just keeps repeating “Invalid two-factor code” even though we all are 100% certain our codes are correct. Using Recovery Codes also does not work. There is just one account (thank goodness) that I have that uses WebAuthN. So I am able to login using that.

If I go to any account and try to disable MFA, it says:
Two-factor authentication has been disabled successfully!
Two-factor authentication: Status Enabled

No one is able to use the system now, EVEN USERS WHO NEVER HAD MFA.
Even users without MFA are getting prompted to enter their verification code, which of course fails.

I’m at a complete loss!
I have a backup of the server, but I think that’s going to be a waste too, because it will point to old YubiKey’s which I no longer have!

Please help! What can I try?

I just tried this:

User.update_all(otp_required_for_login: false, encrypted_otp_secret: nil, encrypted_otp_secret_iv: nil, encrypted_otp_secret_salt: nil, otp_backup_codes: nil)

It says 10 users updated. But it did nothing.

I ran in to something similar for just one user… and was able to fix it in the rails- console.

This is based on: Rails console | GitLab
<from console, get into rails console>

sudo gitlab-rails console
You can look at and compare user profile of working vs none working user with:

u = User.find_by_username(‘<working user id>’)
pp u.attributes

redo for failing user and compare…

To turn OFF MFA in rails you need a ‘function’ So … paste in:

def disable_two_factor!
transaction do
update(
otp_required_for_login: false,
encrypted_otp_secret: nil,
encrypted_otp_secret_iv: nil,
encrypted_otp_secret_salt: nil,
otp_grace_period_started_at: nil,
otp_backup_codes: nil
)
self.u2f_registrations.destroy_all # rubocop: disable DestroyAll
end
end

Then run, for each user that is effected.

user = User.find_by(username: ‘<BadUser>’)
user.disable_two_factor!