Upgrade gitlab-ce-14.7.2 to 14.10.2 (CentOS7) failed

How to Use GitLab Upgrade
1

Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/server.fqdn.json.
This usually happens when your last run of gitlab-ctl reconfigure didn’t complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run sudo gitlab-ctl reconfigure to fix it and try again

File Preformatted text/opt/gitlab/embedded/nodes/server.fqdn reads:
{
“name”: “server.fqdn”
}

leads to:

[root@server.fqdn]# sudo gitlab-ctl reconfigure

Starting Chef Infra Client, version 15.17.4
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - logrotate (0.1.0)
  - postgresql (0.1.0)
  - redis (0.1.0)
  - monitoring (0.1.0)
  - consul (0.1.0)
  - gitaly (0.1.0)
  - praefect (0.1.0)
  - gitlab-kas (0.1.0)
  - registry (0.1.0)
  - gitlab-pages (0.1.0)
  - package (0.1.0)
  - mattermost (0.1.0)
  - letsencrypt (0.1.0)
  - nginx (0.1.0)
  - runit (5.1.3)
  - acme (4.1.3)
  - crond (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: gitlab::default
  * directory[/etc/gitlab] action create (up to date)
  Converging 236 resources
  * directory[/etc/gitlab] action create (up to date)
  * directory[Create /var/opt/gitlab] action create (up to date)
  * directory[Create /var/log/gitlab] action create (up to date)
  * directory[/opt/gitlab/embedded/etc] action create (up to date)
  * template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
  * account[Webserver user and group] action create
    * group[Webserver user and group] action create (up to date)
    * linux_user[Webserver user and group] action create (up to date)
     (up to date)

[…]

Recipe: praefect::disable
  * service[praefect] action nothing (skipped due to action :nothing)
  * runit_service[praefect] action disable
    * ruby_block[disable praefect] action run (skipped due to only_if)
     (up to date)
  * consul_service[praefect] action delete
    * file[/var/opt/gitlab/consul/config.d/praefect-service.json] action delete (up to date)
     (up to date)
Recipe: gitlab-kas::enable
  * directory[/var/opt/gitlab/gitlab-kas] action create (up to date)
  * directory[/var/log/gitlab/gitlab-kas] action create (up to date)
  * directory[/opt/gitlab/etc/gitlab-kas] action create (up to date)
  * ruby_block[websocket TLS termination] action run (skipped due to only_if)
  * version_file[Create version file for Gitlab KAS] action create
    * file[/var/opt/gitlab/gitlab-kas/VERSION] action create (up to date)
     (up to date)
  * file[/var/opt/gitlab/gitlab-kas/authentication_secret_file] action create (up to date)
  * file[/var/opt/gitlab/gitlab-kas/private_api_authentication_secret_file] action create (up to date)
  * file[/var/opt/gitlab/gitlab-kas/redis_password_file] action create (skipped due to only_if)
  * template[/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml] action create (up to date)
  * env_dir[/opt/gitlab/etc/gitlab-kas/env] action create
    * directory[/opt/gitlab/etc/gitlab-kas/env] action create (up to date)
    * file[/opt/gitlab/etc/gitlab-kas/env/SSL_CERT_DIR] action create (up to date)
    * file[/opt/gitlab/etc/gitlab-kas/env/OWN_PRIVATE_API_URL] action create (up to date)
     (up to date)
  * service[gitlab-kas] action nothing (skipped due to action :nothing)
  * runit_service[gitlab-kas] action enable
    * ruby_block[restart_service] action nothing (skipped due to action :nothing)
    * ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
    * ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/sv/gitlab-kas] action create (up to date)
    * template[/opt/gitlab/sv/gitlab-kas/run] action create (up to date)
    * directory[/opt/gitlab/sv/gitlab-kas/log] action create (up to date)
    * directory[/opt/gitlab/sv/gitlab-kas/log/main] action create (up to date)
    * template[/opt/gitlab/sv/gitlab-kas/log/config] action create (up to date)
    * ruby_block[verify_chown_persisted_on_gitlab-kas] action nothing (skipped due to action :nothing)
    * link[/var/log/gitlab/gitlab-kas/config] action create (up to date)
    * template[/opt/gitlab/sv/gitlab-kas/log/run] action create (up to date)
    * directory[/opt/gitlab/sv/gitlab-kas/env] action create (up to date)
    * ruby_block[Delete unmanaged env files for gitlab-kas service] action run (skipped due to only_if)
    * template[/opt/gitlab/sv/gitlab-kas/check] action create (skipped due to only_if)
    * template[/opt/gitlab/sv/gitlab-kas/finish] action create (skipped due to only_if)
    * directory[/opt/gitlab/sv/gitlab-kas/control] action create (up to date)
    * link[/opt/gitlab/init/gitlab-kas] action create (up to date)
    * file[/opt/gitlab/sv/gitlab-kas/down] action nothing (skipped due to action :nothing)
    * directory[/opt/gitlab/service] action create (up to date)
    * link[/opt/gitlab/service/gitlab-kas] action create (up to date)
    * ruby_block[wait for gitlab-kas service socket] action run (skipped due to not_if)
     (up to date)
Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run
      
      ================================================================================
      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
      ================================================================================
      
      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Command execution failed. STDOUT/STDERR suppressed for sensitive resource
      
      Resource Declaration:
      ---------------------
      suppressed sensitive resource output
      
      Compiled Resource:
      ------------------
      suppressed sensitive resource output
      
      System Info:
      ------------
      chef_version=15.17.4
      platform=centos
      platform_version=7.9.2009
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client
      
    
    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55: 
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60: 
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
    
    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=15.17.4
    platform=centos
    platform_version=7.9.2009
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Running handlers complete
Chef Infra Client failed. 0 resources updated in 01 minutes 49 seconds
2

That problem looks very similar to this post.

FWIW I also ran into a similar issue with upgrading to 14.10.2 recently and the resolution from the linked post fixed it.

3 Likes
3

You’ve made my day!
The mentioned two commands

[root@server.fqdn]# sudo gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']
Done.
[root@server.fqdn]# gitlab-ctl reconfigure

and everything runs fine, again.

1 Like