After restoring backup, Gitlab begins to have 500 internal server errors

General
1

ported gitlab to another machine

Gitlab 13.12.3-ee was installed on Ubuntu 18.04 and everything worked fine.

The move was to another distribution, where gitlab was built from source code and the gitlab itself worked well until I restored it from a backup.

System info:

  • CPU: Intel Xeon E-2288G
  • 14Gb RAM
  • Intel SSD DC 4610 480Gb
  • Virtual Machine (KVM, no docker and etc…)
  • 16 CPU thread for vm
  • OS: Linux Gentoo
  • 1 User
  • And no resource limits

bundle exec rake gitlab:env:info RAILS_ENV=production

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.3p183
Gem Version:    3.1.6
Bundler Version:2.2.19
Rake Version:   13.0.3
Redis Version:  6.0.14
Git Version:    2.31.1
Sidekiq Version:5.2.9
Go Version:     go1.16.5 linux/amd64

GitLab information
Version:        13.12.3-ee
Revision:       9d9769ba2ad
Directory:      /opt/gitlab/gitlab
DB Adapter:     PostgreSQL
DB Version:     12.6
URL:            http://192.168.1.5
HTTP Clone URL: http://192.168.1.5/some-group/some-project.git
SSH Clone URL:  git@192.168.1.5/some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        13.18.0
Repository storage paths:
- default:      /var/lib/gitlab/repositories
GitLab Shell path:              /opt/gitlab/gitlab-shell
Git:            /usr/bin/git

bundle exec rake gitlab:check RAILS_ENV=production

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.18.0 ? ... OK (13.18.0)
Running /opt/gitlab/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/5

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Projects have namespace: ... 
dev / winplatform ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.3)
Git version >= 2.31.0 ? ... yes (2.31.1)
Git user has default SSH configuration? ... yes
Active users: ... 1661
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

backup and restore:

gitlab-ctl stop puma
gitlab-ctl stop sidekiq
gitlab-ctl status

gitlab-rake gitlab:backup:create STRATEGY=copy
cp /etc/gitlab/gitlab-secrets.json /root/

rc-service gitlab stop

mv *.tar /opt/gitlab/backups/
chown git:git /opt/gitlab/backups/*.tar

cd /opt/gitlab/gitlab
sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production

I have converted json to yml
cp /root/secrets.yml /opt/gitlab/gitlab/config/

production.log:

secrets.yml:

su git
sh-5.1$ export RUBYOPT=--disable-did_you_mean
sh-5.1$ RAILS_ENV=production bundle exec rails c
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-linux]
 GitLab:       13.12.3-ee (9d9769ba2ad) EE
 GitLab Shell: 13.18.0
 PostgreSQL:   12.6
--------------------------------------------------------------------------------
Loading production environment (Rails 6.0.3.6)
irb(main):001:0> ApplicationSetting.current.reset_runners_registration_token!
Traceback (most recent call last):
        5: from (irb):1
        4: from app/models/concerns/token_authenticatable.rb:53:in `block in add_authentication_token_field'
        3: from app/models/concerns/token_authenticatable_strategies/base.rb:44:in `reset_token!'
        2: from lib/gitlab/database.rb:370:in `transaction'
        1: from lib/gitlab/database.rb:371:in `block in transaction'
OpenSSL::Cipher::CipherError ()

What is the problem?

2

For some reason I can’t edit …
This is gitlab-secrets.json: https://pastebin.com/iQ7BnqVz

4

I have been trying to solve the problem for a very long time, but I cannot.
Many people encounter it after migrating gitlab, but there is no concrete solution even in the wiki…

5

I still cannot get solutions … I see that many who stumble with this problem, issues are created and even find solutions, but their solutions = partially mine.

6

Hi,

I expect your problem is because your Ubuntu installation was an omnibus version, and you attempted to restore it to a source install. As per the docs: Redirecting...

You can only restore a backup to exactly the same version and type (CE/EE) of GitLab on which it was created. The best way to migrate your repositories from one server to another is through backup restore.

I take that to mean, if you backup on omnibus, that means you have to restore it to omnibus. The same if you back up using a source install, then you restore it to a source install. As well as of course, with them both being the same version number, eg: 13.12.3.

I suggest you create an Ubuntu 20.04 installation with omnibus and attempt to restore to it and making sure you have the same version as from the backup, so if 13.12.3, make sure you install 13.12.3 on Ubuntu 20.04. And that will most likely work, unless you are doing something else wrong. Please follow the backup/restore documentation that I linked to make sure.

I have done this before without issues, but then I used omnibus edition installed on a supported distribution.

1 Like