Gitlab-ctl reconfigure doesn't work after gitlab omnibus updated

Updated the gitlab-ee package on an Ubuntu server then ran gitlab-ctl reconfigure as gitlab was throwing a 500. Usually gitlab-ctl reconfigure resolves issues after apt-get update but this time it’s producing the following strange error on the database_migrations step:

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run
      
      ================================================================================
      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
      ================================================================================
      
      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Command execution failed. STDOUT/STDERR suppressed for sensitive resource
      
      Resource Declaration:
      ---------------------
      suppressed sensitive resource output
      
      Compiled Resource:
      ------------------
      suppressed sensitive resource output
      
      System Info:
      ------------
      chef_version=15.17.4
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client
      
    
    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55: 
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60: 
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
    
    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=15.17.4
    platform=ubuntu
    platform_version=20.04
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Running handlers complete
Chef Infra Client failed. 0 resources updated in 21 seconds

I did not make any changes to the configuration and gitlab-ctl does not explain what’s wrong. Update was incremental from v14 but I don’t know which minor version it was.

How can I diagnose and fix this strange error?

$ sudo gitlab-rake gitlab:env:info

System information
System:		Ubuntu 20.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.2.33
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	14.10.0-ee
Revision:	ad109bc62af
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		https://gitlab.xxxxxxx.com
HTTP Clone URL:	https://gitlab.xxxxxxx.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.xxxxxxx.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.25.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Okay, found the fix here:

sudo gitlab-rake db:migrate

and it said it was expecting the project namespaces backfill task to be marked complete but it wasn’t, and to run this command:

sudo gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,‘[null,“up”]’]

Running gitlab-ctl reconfigure then suceeded.

Migdsb, you are my hero. I had exactly the same problem with the same error messages migrating from 14.8.2 to 14.10.0 and your solution worked perfectly for me. I cannot thank you enough!

Keep yourself safe!

Guess I have the same problem, while running in a docker container. I would love to apply the solution. But at container startup - Gitlab runs into the issue and directly shuts down the container. How can I prevent Gitlab from doing this?

I am glad I found this post, as it let us successfully complete our upgrade and reconfigure!

I won’t go over the whole story, but I will back-link to the gitlab.com issue that I found first, where I described what we went through:

In my case, what I do is overwrite the boot command with tail -f /dev/null and once the container is up and running, I run an interactive bash.

In order to interact with gitlab-ctl you must start the service: /opt/gitlab/embedded/bin/runsvdir-start

Once up and running you can run the solution.
First of all, you should run the gitlab-ctl reconfigure command since by overwriting the boot command you are recreating the container.

Once the reconfigure has completed with error, you can confirm that it is the migration error by running the command: gitlab-rake db:migrate

If the error is the command, run the command gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,’[null,“up”]’]" to finish the task.

Once complete, you can run gitlab-ctl reconfigure again and it should work fine.

Hope this can help you.

Thanks, this was exactly what I was looking for and moved me one step ahead. Only that I am getting a very strange error now when attempting to start the Gitlab services -

/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system

Must be something related to our docker environment, I reckon. Test setup is docker-desktop under Windows, I don’t dare to move to production setup that is running docker under ArchLinux. I will have to investigate further.

Would be really cool if that problem could be solved in the next Gitlab version, as we would rather like to focus on our measurement device software development.

Thanks for your advice, it saved my day!

For my case, from 14.9.3 to 14.10.0 on a Docker installation, this worked for me, after migration failed with the command (yours may be a little different):

docker run --detach
–hostname {your_subdomain}
–publish 80:80
–name gitlab
–restart unless-stopped
–volume $HOME/docker/volumes/gitlab/config:/etc/gitlab:Z
–volume $HOME/docker/volumes/gitlab/logs:/var/log/gitlab:Z
–volume $HOME/docker/volumes/gitlab/data:/var/opt/gitlab:Z
–shm-size 256m
gitlab/gitlab-ce:14.10.0-ce.0

docker exec gitlab bash
gitlab-rake db:migrate
gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,’[null,“up”]’]
gitlab-rake db:migrate
gitlab-ctl reconfigure
exit
docker restart gitlab

Hope it helps to fellow colleagues in the Docker ecosystem . . .

That was exactly my case, migrating from 14.8.2 to 14.10.0.

@migdsb provided the right solution, worked like a charm!

Thanks!!!

love u

Thanks !! You saved my job

@migdsb Thanks, saved my day!

On Ubuntu 18.04 I had to type the rake command a little bit different:

gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']

Note the backslash after the null!