Error after APT-Upgrade

How to Use GitLab
1

After upgrading with APT, my gitlab is broken and nothing is working.
In normal scenario i would just backup my data and reinstall, but i am not sure what is happening to my issues and the other data then, and i could not find a howto backup.

The following output gives me a simple:

root@h2740437:/# gitlab-ctl
Traceback (most recent call last):
        29: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
        28: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
        27: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:29:in `<top (required)>'
        26: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `load_files'
        25: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:186:in `each'
        24: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:187:in `block in load_files'
        23: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `load_file'
        22: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:192:in `eval'
        21: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:18:in `load_file'
        20: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:39:in `require'
        19: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:135:in `rescue in require'
        18: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:135:in `require'
        17: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl.rb:17:in `<top (required)>'
        16: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl.rb:17:in `require_relative'
        15: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:2:in `<top (required)>'
        14: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:2:in `require_relative'
        13: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:3:in `<top (required)>'
        12: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:39:in `require'
        11: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
        10: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems.rb:217:in `try_activate'
         9: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems.rb:224:in `rescue in try_activate'
         8: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1440:in `activate'
         7: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `activate_dependencies'
         6: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `each'
         5: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1472:in `block in activate_dependencies'
         4: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1440:in `activate'
         3: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `activate_dependencies'
         2: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1458:in `each'
         1: from /opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/specification.rb:1469:in `block in activate_dependencies'
/opt/gitlab/embedded/lib/ruby/2.5.0/rubygems/dependency.rb:310:in `to_specs': Could not find 'specinfra' (~> 2.72) among 416 total gem(s) (Gem::MissingSpecError)
Checked in 'GEM_PATH=/root/.gem/ruby/2.5.0:/opt/gitlab/embedded/lib/ruby/gems/2.5.0', execute `gem env` for more information

any command started with “gitlab-ctl” gives me the excact same output. even “gitlab-ctl restart” or “gitlab-ctl reconfigure”.

Can anyone tell me what to do, without loosing data and gitdata?
Is it possible to “reinstall” the gems manually?

EDIT:
Backing up the install is also not possible.
Exec ‘gitlab-rake gitlab:backup:create’ gives me following error:

Could not find gitlab-sidekiq-fetcher-0.4.0 in any of the sources
Run `bundle install` to install missing gems.
2

Hi @Sunchezz,

You can try fixing it with the package manager, APT:
sudo apt-get update
sudo apt-get install --reinstall --install-recommends gitlab-ce
sudo gitlab-ctl restart

If that doesn’t work, you can try:
sudo apt-get install -y
sudo dpkg --configure -a
sudo gitlab-ctl reconfigure

If this does not help, can you paste the output of:
sudo gitlab-rake gitlab:check

Best Regards,
Greg M

3

Hi thx for your response.

i fixed it now by manually copying 2 missing *.gemspec files from a gitlab install on another server.
This worked.

I previously made a full VPS restore by the provider with the backup from the day before.
Somehow, magically, these two files (maybe more) seem to be lost trough this restore.

Does --reinstall not override configs and data?

Like i said, EVERY command starting with “gitlab” did not work.
i have tried dpkg --configure, gitlab-rake gitlab:check and gitlab-ctl reconfigure all with the same result. Complaining about missing gems.

If any of the devs want to test this behaviour, remove the files in /gitlab/embedded/lib/ruby/gems/2.5.0/specifications/:
specinfra-2.76.6.gemspec
gitlab-sidekiq-fetcher-0.4.0.gemspec

Thx and good luck! :slight_smile: :wink: