Hi,
My organization is a GitLab enterprise customer so this should probably go directly to gitlab support but I am having some issues contacting them (owner of account is no longer working for the organization, for one thing, for another, our instance of gitlab is down at the moment, which is why I’m posting).
We were running version 13.5.4 of GitLab using the official Docker container. We were affected by NVD - CVE-2021-22192 so I am trying to upgrade to the latest version which is patched against this issue.
First I upgraded to gitlab/gitlab-ee:13.12.12-ee.0 (most recent 13.x version) and then I upgraded to gitlab/gitlab-ee:14.0.11-ee.0 (most recent 14.0 version) and finally I upgraded to gitlab/gitlab-ee:latest. When i say “upgraded to” I mean I changed the image tag in my docker-compose.yml and then ran docker-compose up.
The first two upgrades went fine but the final one gave me the following error:
web_1 | Recipe: gitlab::gitlab-rails
web_1 | * execute[clear the gitlab-rails cache] action run
web_1 | - execute /opt/gitlab/bin/gitlab-rake cache:clear
web_1 | There was an error running gitlab-ctl reconfigure:
web_1 |
web_1 | rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
web_1 | ---- Begin output of "bash" "/tmp/chef-script20211101-30-1isvpsr" ----
web_1 | STDOUT: rake aborted!
web_1 | StandardError: An error has occurred, all later migrations canceled:
web_1 |
web_1 | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
web_1 |
web_1 | Finalize it manualy by running
web_1 |
web_1 | sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
web_1 |
web_1 | For more information, check the documentation
web_1 |
web_1 | https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
web_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
web_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
web_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
web_1 |
web_1 | Caused by:
web_1 | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
web_1 |
web_1 | Finalize it manualy by running
web_1 |
web_1 | sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
web_1 |
web_1 | For more information, check the documentation
web_1 |
web_1 | https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
web_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
web_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
web_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
web_1 | Tasks: TOP => db:migrate
web_1 | (See full trace by running task with --trace)
web_1 | == 20210622045705 FinalizeEventsBigintConversion: migrating ===================
web_1 | STDERR:
web_1 | ---- End output of "bash" "/tmp/chef-script20211101-30-1isvpsr" ----
web_1 | Ran "bash" "/tmp/chef-script20211101-30-1isvpsr" returned 1
web_1 |
web_1 |
web_1 | Running handlers:
web_1 | Running handlers complete
web_1 | Chef Infra Client failed. 131 resources updated in 01 minutes 24 seconds
Can you please let me know how to fix this?
Also, there seems to be an issue with gitlab/gitlab-ee:latest.
The version of exiftool in that image is 11.70 which is the same version it was in the image I was running before the incident. According to NVD - CVE-2021-22204 (see “Known Affected Software Configurations” section) this is an affected version of exiftool and the image should have the latest version (12.3.4 released 10/27/2021). Can you push out a new image that has a patched version of exiftool?
Thank you.