Upgrade from 16.9.1 to 16.11.4

I have launched the version update and when I run the post-installation script I am getting this error.

It won’t let me start the server or go back to the previous version.

Dumping PostgreSQL database gitlabhq_production ... [DONE]
2024-06-12 14:16:38 UTC -- Dumping database ... done
2024-06-12 14:16:38 UTC -- Dumping repositories ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping uploads ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping builds ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping artifacts ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping pages ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping lfs objects ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping terraform states ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping container registry images ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping packages ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Dumping ci secure files ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Creating backup archive: 1718201794_2024_06_12_16.9.1_gitlab_backup.tar ... 
2024-06-12 14:16:38 UTC -- Creating backup archive: 1718201794_2024_06_12_16.9.1_gitlab_backup.tar ... done
2024-06-12 14:16:38 UTC -- Uploading backup archive to remote storage  ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Deleting old backups ... [SKIPPED]
2024-06-12 14:16:38 UTC -- Deleting tar staging files ... 
2024-06-12 14:16:38 UTC -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2024-06-12 14:16:38 UTC -- Cleaning up /var/opt/gitlab/backups/db
2024-06-12 14:16:38 UTC -- Deleting tar staging files ... done
2024-06-12 14:16:38 UTC -- Deleting backups/tmp ... 
2024-06-12 14:16:38 UTC -- Deleting backups/tmp ... done
2024-06-12 14:16:38 UTC -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
2024-06-12 14:16:38 UTC -- Backup 1718201794_2024_06_12_16.9.1 is done.
2024-06-12 14:16:38 UTC -- Deleting backup and restore PID file ... done
gitlab preinstall: Automatically backing up /etc/gitlab
Running configuration backup
Creating configuration backup archive: gitlab_config_1718201798_2024_06_12.tar
/etc/gitlab/
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/ssl/
/etc/gitlab/ssl/git.gutierrezti.es.crt
/etc/gitlab/ssl/git.gutierrezti.es.key-staging
/etc/gitlab/ssl/git.gutierrezti.es.key
/etc/gitlab/ssl/git.gutierrezti.es.crt-staging
/etc/gitlab/gitlab.rb
/etc/gitlab/trusted-certs/
Configuration backup archive complete: /etc/gitlab/config_backup/gitlab_config_1718201798_2024_06_12.tar
Keeping all older configuration backups
Desempaquetando gitlab-ce (16.11.4-ce.0) sobre (16.9.1-ce.0) ...
Configurando gitlab-ce (16.11.4-ce.0) ...
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/specification.rb:2317:in `raise_if_conflicts': Unable to activate chef-18.3.0, because chef-utils-18.4.2 conflicts with chef-utils (= 18.3.0) (Gem::ConflictError)
	from /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/specification.rb:1446:in `activate'
	from /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems.rb:205:in `rescue in try_activate'
	from /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems.rb:198:in `try_activate'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:139:in `block in require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:139:in `synchronize'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:139:in `rescue in require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from /opt/gitlab/embedded/cookbooks/package/libraries/settings_dsl.rb:19:in `<top (required)>'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `require_relative'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `<top (required)>'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /opt/gitlab/embedded/service/omnibus-ctl/check_config.rb:18:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:192:in `eval'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:192:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:187:in `block in load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:186:in `each'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:186:in `load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/bin/omnibus-ctl:29:in `<top (required)>'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/specification.rb:2317:in `raise_if_conflicts': Unable to activate chef-18.3.0, because chef-utils-18.4.2 conflicts with chef-utils (= 18.3.0) (Gem::ConflictError)
	from /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/specification.rb:1446:in `activate'
	from /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems.rb:199:in `try_activate'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:139:in `block in require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:139:in `synchronize'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:139:in `rescue in require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from /opt/gitlab/embedded/cookbooks/package/libraries/settings_dsl.rb:19:in `<top (required)>'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `require_relative'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `<top (required)>'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /opt/gitlab/embedded/service/omnibus-ctl/check_config.rb:18:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:192:in `eval'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:192:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:187:in `block in load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:186:in `each'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:186:in `load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/bin/omnibus-ctl:29:in `<top (required)>'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- chef/json_compat (LoadError)
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /opt/gitlab/embedded/cookbooks/package/libraries/settings_dsl.rb:19:in `<top (required)>'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `require_relative'
	from /opt/gitlab/embedded/cookbooks/package/libraries/deprecations.rb:3:in `<top (required)>'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /opt/gitlab/embedded/service/omnibus-ctl/check_config.rb:18:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:192:in `eval'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:192:in `load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:187:in `block in load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:186:in `each'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:186:in `load_files'
	from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/bin/omnibus-ctl:29:in `<top (required)>'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `load'
	from /opt/gitlab/embedded/bin/omnibus-ctl:25:in `<main>'
dpkg: error al procesar el paquete gitlab-ce (--configure):
 el subproceso instalado paquete gitlab-ce script post-installation devolvió el código de salida de error 1
Se encontraron errores al procesar:
 gitlab-ce

What is happening?

Best regards and thanks for your help

First I am hoping that you have a full backup (and the /etc/gitlab secrets) in case of the need to re-install. Unfortunately it is not the first time that upgrade paths are broken due to borked dependency versions.

Another issue also due to mismatched dependency versions (with a different error message) happened some time ago: Failure when upgrading to 16.9.7 (CE and EE editions) (#8532) · Issues · GitLab.org / omnibus-gitlab · GitLab

My two cents and no express or implied warranties - I am not affiliated with GitLab support team:

Install Specific Gem Versions: You can try installing the specific versions required. Use the following commands:

sudo /opt/gitlab/embedded/bin/gem uninstall chef-utils -v 18.4.2
sudo /opt/gitlab/embedded/bin/gem install chef -v 18.3.0
sudo /opt/gitlab/embedded/bin/gem install chef-utils -v 18.3.0

Then reconfigure: sudo dpkg --configure gitlab-ce

Good luck! And please raise awareness as an issue: Issues · GitLab.org / omnibus-gitlab · GitLab

1 Like

Hello Silva:

With your help and the indicated commands I have managed to get it to boot. The problem is that now I get a 502 error and message waiting for gitlab to start.

I have checked the services and everything is working.
When I run the command
sudo gitlab-rake gitlab:check --trace
I get the following:
** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
rake aborted!
ArgumentError: Data consistency is already set
/opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in data_consistency' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in block in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/lazy_load_hooks.rb:92:in block in execute_hook' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/lazy_load_hooks.rb:85:in with_execution_control’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/lazy_load_hooks.rb:90:in execute_hook' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/lazy_load_hooks.rb:76:in block in run_load_hooks’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/lazy_load_hooks.rb:75:in each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/lazy_load_hooks.rb:75:in run_load_hooks’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/application/finisher.rb:87:in block in <module:Finisher>' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:32:in instance_exec’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:32:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:61:in block in run_initializers’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:228:in block in tsort_each' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:350:in block (2 levels) in each_strongly_connected_component’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:422:in block (2 levels) in each_strongly_connected_component_from’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:431:in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:421:in block in each_strongly_connected_component_from’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in tsort_each_child’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in call' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in each_strongly_connected_component_from’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:421:in block in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in tsort_each_child' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in call’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:421:in block in each_strongly_connected_component_from’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in tsort_each_child’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in call' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in each_strongly_connected_component_from’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:421:in block in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:50:in tsort_each_child' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in call’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:415:in each_strongly_connected_component_from' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:349:in block in each_strongly_connected_component’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:347:in each' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:347:in call’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:347:in each_strongly_connected_component' /opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:226:in tsort_each’
/opt/gitlab/embedded/lib/ruby/3.1.0/tsort.rb:205:in tsort_each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/initializable.rb:60:in run_initializers’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/application.rb:372:in initialize!' /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/zeitwerk-2.6.7/lib/zeitwerk/kernel.rb:38:in require' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/application.rb:348:in require_environment!’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/railties-7.0.8.1/lib/rails/application.rb:506:in block in run_tasks_blocks' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in block in execute’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in execute’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:24:in execute' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in block in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in invoke' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in block in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in block in execute' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in execute' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:24:in execute’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in block in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:243:in block in invoke_prerequisites’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:241:in invoke_prerequisites’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:218:in block in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in invoke’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in invoke_task' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block in top_level’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in run_with_threads' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in top_level’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in block in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in standard_exception_handling’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>’
/opt/gitlab/embedded/bin/rake:25:in load' /opt/gitlab/embedded/bin/rake:25:in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/cli/exec.rb:58:in load' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/cli/exec.rb:58:in kernel_load’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/cli/exec.rb:23:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/cli.rb:451:in exec’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/vendor/thor/lib/thor/command.rb:28:in run' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/vendor/thor/lib/thor.rb:527:in dispatch' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/cli.rb:34:in dispatch’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/vendor/thor/lib/thor/base.rb:584:in start' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/cli.rb:28:in start’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/exe/bundle:28:in block in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/lib/bundler/friendly_errors.rb:117:in with_friendly_errors’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.8/exe/bundle:20:in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in load’
/opt/gitlab/embedded/bin/bundle:25:in `’
Tasks: TOP => environment

Greetings and thanks for your help

More info:

Recipe: gitlab::database_migrations

  • ruby_block[check remote PG version] action nothing (skipped due to action :nothing)

  • rails_migration[gitlab-rails] action run[2024-06-13T12:35:21+02:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.

    • bash_hide_env[migrate gitlab-rails database] action run
      [execute] rake aborted!
      ArgumentError: Data consistency is already set
      /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in data_consistency' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in block in <top (required)>’
      /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
      internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' /opt/gitlab/embedded/bin/bundle:25:in load’
      /opt/gitlab/embedded/bin/bundle:25:in `’
      Tasks: TOP => gitlab:db:configure => environment
      (See full trace by running task with --trace)

      ================================================================================
      Error executing action run on resource ‘bash_hide_env[migrate gitlab-rails database]’

      Mixlib::ShellOut::ShellCommandFailed

      Expected process to exit with [0], but received ‘1’
      ---- Begin output of “bash” ----
      STDOUT: rake aborted!
      ArgumentError: Data consistency is already set
      /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in data_consistency' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in block in <top (required)>’
      /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
      internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' /opt/gitlab/embedded/bin/bundle:25:in load’
      /opt/gitlab/embedded/bin/bundle:25:in `’
      Tasks: TOP => gitlab:db:configure => environment
      (See full trace by running task with --trace)
      STDERR:
      ---- End output of “bash” ----
      Ran “bash” returned 1

      Cookbook Trace: (most recent call first)

      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file’

      Resource Declaration:

      In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb

      20: bash_hide_env “migrate #{new_resource.name} database” do
      21: code <<-EOH
      22: set -e
      23: log_file=“#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log”
      24: umask 077
      25: /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file}
      26: STATUS=${PIPESTATUS[0]}
      27: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
      28: echo $STATUS > #{new_resource.helper.db_migrate_status_file}
      29: exit $STATUS
      30: EOH
      31:
      32: environment new_resource.environment if new_resource.property_is_set?(:environment)
      33: new_resource.dependent_services.each do |svc|
      34: notifies :restart, svc, :immediately
      35: end
      36:
      37: not_if { new_resource.helper.migrated? }
      38: sensitive false
      39: end
      40: end

      Compiled Resource:

      Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file’

      bash_hide_env(“migrate gitlab-rails database”) do
      action [:run]
      default_guard_interpreter :default
      interpreter “bash”
      declared_type :bash_hide_env
      cookbook_name “gitlab”
      recipe_name “database_migrations”
      code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n chown git:git ${log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-c6697a9cd49e1cfa45cc31b1ecdb90ae-83a2fa9fae0\n exit $STATUS\n"
      environment “sensitive value suppressed
      not_if { #code block }
      end

      System Info:

      chef_version=18.3.0
      platform=ubuntu
      platform_version=22.04
      ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/cinc-client
      executable=/opt/gitlab/embedded/bin/cinc-client

    ================================================================================
    Error executing action run on resource ‘rails_migration[gitlab-rails]’

    Mixlib::ShellOut::ShellCommandFailed

    bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” ----
    STDOUT: rake aborted!
    ArgumentError: Data consistency is already set
    /opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in data_consistency' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in block in <top (required)>’
    /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
    internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' /opt/gitlab/embedded/bin/bundle:25:in load’
    /opt/gitlab/embedded/bin/bundle:25:in `’
    Tasks: TOP => gitlab:db:configure => environment
    (See full trace by running task with --trace)
    STDERR:
    ---- End output of “bash” ----
    Ran “bash” returned 1

    Cookbook Trace: (most recent call first)

    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file’

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

    51: rails_migration “gitlab-rails” do
    52: rake_task ‘gitlab:db:configure’
    53: logfile_prefix ‘gitlab-rails-db-migrate’
    54: helper migration_helper
    55:
    56: environment env_variables
    57: dependent_services dependent_services
    58: notifies :run, “execute[clear the gitlab-rails cache]”, :immediately
    59: notifies :run, “ruby_block[check remote PG version]”, :immediately
    60:
    61: only_if { migration_helper.attributes_node[‘auto_migrate’] }
    62: end

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file’

    rails_migration(“gitlab-rails”) do
    action [:run]
    default_guard_interpreter :default
    declared_type :rails_migration
    cookbook_name “gitlab”
    recipe_name “database_migrations”
    rake_task “gitlab:db:configure”
    logfile_prefix “gitlab-rails-db-migrate”
    helper “sensitive value suppressed
    environment “sensitive value suppressed
    dependent_services [“runit_service[puma]”, “sidekiq_service[sidekiq]”]
    only_if { #code block }
    end

    System Info:

    chef_version=18.3.0
    platform=ubuntu
    platform_version=22.04
    ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/cinc-client
    executable=/opt/gitlab/embedded/bin/cinc-client

[2024-06-13T12:35:36+02:00] INFO: Running queued delayed notifications before re-raising exception

Running handlers:
[2024-06-13T12:35:36+02:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
ArgumentError: Data consistency is already set
/opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in data_consistency' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in block in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' /opt/gitlab/embedded/bin/bundle:25:in load’
/opt/gitlab/embedded/bin/bundle:25:in `’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” ----
Ran “bash” returned 1

Running handlers complete
[2024-06-13T12:35:36+02:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 23 seconds
[2024-06-13T12:35:36+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2024-06-13T12:35:36+02:00] FATAL: ---------------------------------------------------------------------------------------
[2024-06-13T12:35:36+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-06-13T12:35:36+02:00] FATAL: ---------------------------------------------------------------------------------------
[2024-06-13T12:35:36+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” ----
STDOUT: rake aborted!
ArgumentError: Data consistency is already set
/opt/gitlab/embedded/service/gitlab-rails/app/workers/concerns/worker_attributes.rb:96:in data_consistency' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mailer_retries.rb:6:in block in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' <internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in require’
internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb:37:in require' /opt/gitlab/embedded/bin/bundle:25:in load’
/opt/gitlab/embedded/bin/bundle:25:in `’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” ----
Ran “bash” returned 1

Then move the /opt/gitlab/embedded/nodes/*.json files out of the way and run gitlab-ctl reconfigure.

And please do make a GitLab issue with the findings and leave a comment here.

If I move the file it does not even start the reconfigure. I have put it back to be in the same step.

I am still looking to see how I can fix this.

Thanks for your help

I think that then you should file an issue as that is the only way to get more official attention from GitLab developers. I am holding upgrading my instance as I saw your post about the problems.

If the reconfigure is no longer blocking apt, you can probably try using an earlier version

apt install gitlab-ce=16.11.3-ce.0 ; apt-mark hold gitlab-ce