500 on login after restoring backup and upgrading

Hey there,

I recently installed gitlab on a new server, and restored a backup (I made sure to install the same version of the backup - 10.8.7). I got a 500 error when I’d log in.

I read a few issues and found some people resolved it by proceeding to run the upgrade, so I then ran the upgrade, which seemed to go smoothly. I still get a 500 error.

I am using username/password with no LDAP or Oauth

I ran the following commands:

yum install gitlab-ce-10.8.7
vim /etc/gitlab/gitlab.rb # I changed the url to https
gitlab-rake gitlab:backup:restore BACKUP=<My backup file>
gitlab-ctl reconfigure
gitlab-ctl restart

That seemed to go smoothly, but like I said, I could not log in

yum update
gitlab-ctl upgrade
gitlab-ctl reconfigure
gitlab-ctl restart

This also seemed to go smoothly, but alas, still a 500

I’ll be happy to provide any other logs to help determine the problem/solution

After doing some digging, I found the following in my production.log:

Completed 500 Internal Server Error in 30ms (ActiveRecord: 2.3ms)
Sending event 8cb5e56bc9e84b7585903772ae3a6602 to Sentry

OpenSSL::Cipher::CipherError (bad decrypt):

app/controllers/sessions_controller.rb:189:in `valid_otp_attempt?'
app/controllers/concerns/authenticates_with_two_factor.rb:59:in `authenticate_with_two_factor_via_otp'
app/controllers/concerns/authenticates_with_two_factor.rb:48:in `authenticate_with_two_factor'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:19:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

So with some Googling, I figure this might be an issue with a .secret file not being moved during import/export.

(Updating my own thread for others’ sake who may find this later)

From what I’ve read, the .secret file is old news. I’ve found a relevant looking file in /etc/gitlab/gitlab-secrets.json.

Comparing the old server’s file contents with the new one, I can see that the contents are different.

So I’ve copied the file over (obviously keeping a backup of the one generated by the installation) - no good.


So I followed the following KB article to just disable 2FA for everyone. Now I can log in.

According to this section of the backup/restore docs, I should probably also delete my ci_variables in my database so I can use the runner.