I just tried to upgrade to 13.10 from 13.4.2 with ‘docker-compose pull’ and the following migration fails when restarting with ‘docker compose up’. I could upgrade to 13.5.7 but the next step upgrade to 13.6.6 fails with the same output. It seems I don’t have columns “verification_retry_at” of relation “terraform_states”
gitlab_1 | Running handlers:
gitlab_1 | There was an error running gitlab-ctl reconfigure:
gitlab_1 |
gitlab_1 | bash[migrate gitlab-rails database] (gitlab::database_migrations line 70) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
gitlab_1 | ---- Begin output of "bash" "/tmp/chef-script20210403-30-119f5nr" ----
gitlab_1 | STDOUT: rake aborted!
gitlab_1 | StandardError: An error has occurred, all later migrations canceled:
gitlab_1 |
gitlab_1 | PG::UndefinedColumn: ERROR: column "verification_retry_at" of relation "terraform_states" does not exist
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201103013242_remove_terraform_state_verification_columns.rb:12:in `block in up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201103013242_remove_terraform_state_verification_columns.rb:11:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 |
gitlab_1 | Caused by:
gitlab_1 | ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "verification_retry_at" of relation "terraform_states" does not exist
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201103013242_remove_terraform_state_verification_columns.rb:12:in `block in up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201103013242_remove_terraform_state_verification_columns.rb:11:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 |
gitlab_1 | Caused by:
gitlab_1 | PG::UndefinedColumn: ERROR: column "verification_retry_at" of relation "terraform_states" does not exist
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201103013242_remove_terraform_state_verification_columns.rb:12:in `block in up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20201103013242_remove_terraform_state_verification_columns.rb:11:in `up'
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:23:in `<main>'
gitlab_1 | Tasks: TOP => db:migrate
gitlab_1 | (See full trace by running task with --trace)
gitlab_1 | == 20201103013242 RemoveTerraformStateVerificationColumns: migrating ==========
gitlab_1 | -- transaction()
gitlab_1 | -- remove_column(:terraform_states, :verification_retry_at, :datetime_with_timezone)
gitlab_1 | STDERR:
gitlab_1 | ---- End output of "bash" "/tmp/chef-script20210403-30-119f5nr" ----
gitlab_1 | Ran "bash" "/tmp/chef-script20210403-30-119f5nr" returned 1
gitlab_1 |
gitlab_1 | Running handlers complete
gitlab_1 | Chef Infra Client failed. 9 resources updated in 03 minutes 07 seconds
When the migrations fail, docker-compose up stops and gitlab doesn’t even start further. Any idea what to do? As it doesn’t seem like others had this error, I’m not sure what went wrong as updates have been straightforward for the last 2 years.