Upgrade to 10.4 failed server error 500

How to Use GitLab Upgrade
1

Hi,

On debian 8 server, I have upgraded package gitlab-ce from 8.17 to 10.4.
But after upgrade to 10.4.4, I got server error 500.
After digging on forums, I understood that I should have updated by step.
But now, it’s too late.

I have successfully updated postgre to version 9.6.5.
And I have also updated gitlab.rb config file to 10.4 version.

But it looks like there is a failure in database migration.

Can someone help me to solve the problem ?
Can I go back to previous version ?
Thank you.

gitlab-ctl reconfigure returned an error:

Recipe: gitlab::database_migrations

  • bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
    StandardError: An error has occurred, all later migrations canceled:

          PG::DuplicateTable: ERROR:  relation "chat_teams" already exists
      : CREATE TABLE "chat_teams" ("id" serial primary key, "namespace_id" integer NOT NULL, "team_id" character varying, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL) 
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in `change'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      
      Caused by:
      ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR:  relation "chat_teams" already exists
      : CREATE TABLE "chat_teams" ("id" serial primary key, "namespace_id" integer NOT NULL, "team_id" character varying, "name" character varying, "created_at" timestamp NOT NULL, "updated_at" timestamp NOT NULL) 
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in `change'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      
      Caused by:
      PG::DuplicateTable: ERROR:  relation "chat_teams" already exists
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in `change'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      Tasks: TOP => db:migrate
      (See full trace by running task with --trace)
      == 20170120131253 CreateChatTeams: migrating ==================================
      -- create_table(:chat_teams)

    ================================================================================
    Error executing action run on resource ‘bash[migrate gitlab-rails database]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” “/tmp/chef-script20180218-1945-1gfonak” ----
    STDOUT: rake aborted!
    StandardError: An error has occurred, all later migrations canceled:

    PG::DuplicateTable: ERROR: relation “chat_teams” already exists
    : CREATE TABLE “chat_teams” (“id” serial primary key, “namespace_id” integer NOT NULL, “team_id” character varying, “name” character varying, “created_at” timestamp NOT NULL, “updated_at” timestamp NOT NULL)
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in change' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

    Caused by:
    ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR: relation “chat_teams” already exists
    : CREATE TABLE “chat_teams” (“id” serial primary key, “namespace_id” integer NOT NULL, “team_id” character varying, “name” character varying, “created_at” timestamp NOT NULL, “updated_at” timestamp NOT NULL)
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in change' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

    Caused by:
    PG::DuplicateTable: ERROR: relation “chat_teams” already exists
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in change' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    == 20170120131253 CreateChatTeams: migrating ==================================
    – create_table(:chat_teams)
    STDERR:
    ---- End output of “bash” “/tmp/chef-script20180218-1945-1gfonak” ----
    Ran “bash” “/tmp/chef-script20180218-1945-1gfonak” returned 1

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

    51: bash “migrate gitlab-rails database” do
    52: code <<-EOH
    53: set -e
    54: log_file=“#{node[‘gitlab’][‘gitlab-rails’][‘log_directory’]}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log”
    55: umask 077
    56: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
    57: STATUS=${PIPESTATUS[0]}
    58: echo $STATUS > #{db_migrate_status_file}
    59: exit $STATUS
    60: EOH
    61: environment env_variables unless env_variables.empty?
    62: notifies :run, “execute[clear the gitlab-rails cache]”, :immediately
    63: dependent_services.each do |svc|
    64: notifies :restart, svc, :immediately
    65: end
    66: not_if “(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)”
    67: only_if { node[‘gitlab’][‘gitlab-rails’][‘auto_migrate’] }
    68: end

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file’

    bash(“migrate gitlab-rails database”) do
    action [:run]
    retries 0
    retry_delay 2
    default_guard_interpreter :default
    command “migrate gitlab-rails database”
    backup 5
    returns 0
    code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-1d22950\n exit $STATUS\n"
    interpreter “bash”
    declared_type :bash
    cookbook_name “gitlab”
    recipe_name “database_migrations”
    not_if “(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-1d22950) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-1d22950 | grep -Fx 0)”
    only_if { #code block }
    end

    Platform:

    x86_64-linux

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20180218-1945-1gfonak” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::DuplicateTable: ERROR: relation “chat_teams” already exists
: CREATE TABLE “chat_teams” (“id” serial primary key, “namespace_id” integer NOT NULL, “team_id” character varying, “name” character varying, “created_at” timestamp NOT NULL, “updated_at” timestamp NOT NULL)
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in change' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR: relation “chat_teams” already exists
: CREATE TABLE “chat_teams” (“id” serial primary key, “namespace_id” integer NOT NULL, “team_id” character varying, “name” character varying, “created_at” timestamp NOT NULL, “updated_at” timestamp NOT NULL)
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in change' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’

Caused by:
PG::DuplicateTable: ERROR: relation “chat_teams” already exists
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170120131253_create_chat_teams.rb:11:in change' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20170120131253 CreateChatTeams: migrating ==================================
– create_table(:chat_teams)
STDERR:
---- End output of “bash” “/tmp/chef-script20180218-1945-1gfonak” ----
Ran “bash” “/tmp/chef-script20180218-1945-1gfonak” returned 1

2

Solved by doing:
gitlab-rake db:migrate:status returned that some items were down
gitlab-rails dbconsole provided me access to the psql database prompt
Into psql prompt:
select * from chat_teams; to check the table was empty
drop table chat_teams; to delete the table
After exiting the psql prompt:
gitlab-rake db:migrate to try again the db migration and this time succeed
gitlab-rake db:migrate:status returned everything up
gitlab-ctl reconfigure this time succeed
gitlab-ctl restart and my server is now up again

I hope it could help someone.