Gitlab-ctl reconfigure fails with 12.0.2-ce.0 upgrade 'PG::DuplicateColumn: ERROR: column "deployment_events" of relation "services" already exists'

Hi,

I have a issue to upgrade GitLab to version 12.0.2-ce.0. I’m using docker to manage GitLab because I have no skills in the deployment of a complex system.

In the log of docker, I have the following messages:

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 53) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20191108-29-10wjii4" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::DuplicateColumn: ERROR:  column "deployment_events" of relation "services" already exists
: ALTER TABLE "services" ADD "deployment_events" boolean DEFAULT NULL
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:425:in `block (2 levels) in add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:421:in `block in add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:258:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:420:in `add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190426180107_add_deployment_events_to_services.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR:  column "deployment_events" of relation "services" already exists
: ALTER TABLE "services" ADD "deployment_events" boolean DEFAULT NULL
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:425:in `block (2 levels) in add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:421:in `block in add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:258:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:420:in `add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190426180107_add_deployment_events_to_services.rb:11:in `up'
Running handlers complete
Chef Client failed. 9 resources updated in 27 seconds
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::DuplicateColumn: ERROR:  column "deployment_events" of relation "services" already exists
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:425:in `block (2 levels) in add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:421:in `block in add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:258:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:420:in `add_column_with_default'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190426180107_add_deployment_events_to_services.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20190426180107 AddDeploymentEventsToServices: migrating ====================
-- transaction_open?()
   -> 0.0000s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- transaction()
-- add_column(:services, :deployment_events, :boolean, {:default=>nil})
-- execute("RESET ALL")
   -> 0.0004s
STDERR: 
---- End output of "bash"  "/tmp/chef-script20191108-29-10wjii4" ----
Ran "bash"  "/tmp/chef-script20191108-29-10wjii4" returned 1

I don’t want to break anything more. Is this forum a good place to discuss this issue or do I need to submit an issue to GitLab?

Using gitlab/gitlab-ce:latest, I got the following error:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              Your current database version is too old to be migrated. You should upgrade to GitLab 11.11.0 before moving to this version. Please see https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/migrate/schema_check.rake:13:in `block in <top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => db:migrate => schema_version_check
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of "bash"  "/tmp/chef-script20191108-25-1rxvxmo" ----
    STDOUT: rake aborted!
    Your current database version is too old to be migrated. You should upgrade to GitLab 11.11.0 before moving to this version. Please see https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/migrate/schema_check.rake:13:in `block in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => db:migrate => schema_version_check
    (See full trace by running task with --trace)
    STDERR: 
    ---- End output of "bash"  "/tmp/chef-script20191108-25-1rxvxmo" ----
    Ran "bash"  "/tmp/chef-script20191108-25-1rxvxmo" returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
    
     54: bash "migrate gitlab-rails database" do
     55:   code <<-EOH
     56:     set -e
     57:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
     58:     umask 077
     59:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
     60:     STATUS=${PIPESTATUS[0]}
     61:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
     62:     echo $STATUS > #{db_migrate_status_file}
     63:     exit $STATUS
     64:   EOH
     65:   environment env_variables unless env_variables.empty?
     66:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     67:   dependent_services.each do |svc|
     68:     notifies :restart, svc, :immediately
     69:   end
     70:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
     71:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
     72: end
     73: 
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:54:in `from_file'
    
    bash("migrate gitlab-rails database") do
      action [:run]
      default_guard_interpreter :default
      command nil
      backup 5
      interpreter "bash"
      declared_type :bash
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-393a5bdafa2\n    exit $STATUS\n"
      domain nil
      user nil
      not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-393a5bdafa2) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-393a5bdafa2 | grep -Fx 0)"
      only_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=14.13.11
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    - execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: <Dynamically Defined Resource>
  * service[gitaly] action restart
    - restart service service[gitaly]
Recipe: gitaly::enable
  * runit_service[gitaly] action hup
    - send hup to runit_service[gitaly]

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20191108-25-1rxvxmo" ----
STDOUT: rake aborted!
Your current database version is too old to be migrated. You should upgrade to GitLab 11.11.0 before moving to this version. Please see https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/migrate/schema_check.rake:13:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate => schema_version_check
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20191108-25-1rxvxmo" ----
Running handlers complete
Ran "bash"  "/tmp/chef-script20191108-25-1rxvxmo" returned 1

Chef Client failed. 125 resources updated in 01 minutes 19 seconds

I find a way to fix it. I had to edit manually the database as in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4085.

First, connect to the database:

sudo gitlab-psql -d gitlabhq_production

Second, delete the column:

ALTER TABLE services DROP COLUMN deployment_events;