Fails to install updates

shaman.k · September 19, 2022, 2:57pm

The GitLab control panel indicates that an update is available for the GitLab software, however the update fails to install.

Currently version 15.2.0 is running, the latest version (15.2.2) resolves 18 security issues and 6 bugs; thus should be installed to address these issues.

Error shown when updating via apt is:

Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/git-lab.adamsonsystems.ca.json.

This usually happens when your last run of gitlab-ctl reconfigure didn’t complete successfully.

This file is used to check if any of the unsupported configurations are enabled,

and hence require a working reconfigure before upgrading.

Please run sudo gitlab-ctl reconfigure to fix it and try again.

dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_15.2.2-ce.0_amd64.deb (–unpack):

new gitlab-ce package pre-installation script subprocess returned error exit status 1

Error shown when running gitlab-ctl reconfigure is:

  ================================================================================
  Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
  ================================================================================

  Mixlib::ShellOut::ShellCommandFailed
  ------------------------------------
  Command execution failed. STDOUT/STDERR suppressed for sensitive resource

  Cookbook Trace: (most recent call first)
  ----------------------------------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'

  Resource Declaration:
  ---------------------
  suppressed sensitive resource output

  Compiled Resource:
  ------------------
  suppressed sensitive resource output

  System Info:
  ------------
  chef_version=17.10.0
  platform=ubuntu
  platform_version=18.04
  ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
  program_name=/opt/gitlab/embedded/bin/cinc-client
  executable=/opt/gitlab/embedded/bin/cinc-client


================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 51: rails_migration "gitlab-rails" do
 52:   rake_task 'gitlab:db:configure'
 53:   logfile_prefix 'gitlab-rails-db-migrate'
 54:   helper migration_helper
 55:
 56:   environment env_variables
 57:   dependent_services dependent_services
 58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 59:   notifies :run, "ruby_block[check remote PG version]", :immediately
 60:
 61:   only_if { migration_helper.attributes_node['auto_migrate'] }
 62: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

rails_migration("gitlab-rails") do
  action [:run]
  default_guard_interpreter :default
  declared_type :rails_migration
  cookbook_name "gitlab"
  recipe_name "database_migrations"
  rake_task "gitlab:db:configure"
  logfile_prefix "gitlab-rails-db-migrate"
  helper "*sensitive value suppressed*"
  environment "*sensitive value suppressed*"
  dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
  only_if { #code block }
end

System Info:
------------
chef_version=17.10.0
platform=ubuntu
platform_version=18.04
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client

[2022-08-05T14:00:56-04:00] INFO: Running queued delayed notifications before re-raising exception

Running handlers:
[2022-08-05T14:00:56-04:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Running ``gitlab-rake db:migrate results in:

main: == 20220707075300 RescheduleBackfillImportedIssueSearchData: migrating ========

rake aborted!

StandardError: An error has occurred, all later migrations canceled:

PG::CheckViolation: ERROR: new row for relation “batched_background_migrations” violates check constraint “check_max_value_in_range”

DETAIL: Failing row contains (47, 2022-08-05 18:15:44.377296+00, 2022-08-05 18:15:44.377296+00, 14, 11, 50000, 1000, 120, 1, BackfillImportedIssueSearchData, PrimaryKeyBatchingStrategy, issues, id, , 7, 100, null, 2022-08-05 18:15:44.37364+00, null, gitlab_main).

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in class:ConnectionProxy’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:331:in `block in transaction’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:330:in `transaction’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:132:in `queue_batched_background_migration’

/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20220707075300_reschedule_backfill_imported_issue_search_data.rb:21:in `up’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration’

/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction’

/opt/gitlab/embedded/bin/bundle:23:in `load’

/opt/gitlab/embedded/bin/bundle:23:in `’

Caused by:

ActiveRecord::StatementInvalid: PG::CheckViolation: ERROR: new row for relation “batched_background_migrations” violates check constraint “check_max_value_in_range”