Gitlab reconfigure Update crash of today

I updated gitlab today using apt udpate/upgrade and then ran gitlab-ctl reconfigure as root. Here is the error I got:

Recipe: gitlab::database_migrations

  • bash[migrate gitlab-rails database] action run
    [execute] == 20200504191813 AddFingerprintToEvents: migrating ===========================
    – column_exists?(:events, :fingerprint)
    -> 0.0030s
    – add_column(:events, :fingerprint, :binary)
    -> 0.0095s
    – transaction_open?()
    -> 0.0000s
    – execute(“ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n”)
    -> 0.0100s
    – execute(“ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;”)
    -> 0.0162s
    == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

          == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
          -- transaction_open?()
             -> 0.0000s
          -- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
             -> 0.0113s
          -- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
             -> 0.0216s
          == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 
          
          == 20200515152649 EnableBtreeGistExtension: migrating =========================
          -- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
          rake aborted!
          StandardError: An error has occurred, this and all later migrations canceled:
          
          PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
          HINT:  Must be superuser to create this extension.
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          
          Caused by:
          ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
          HINT:  Must be superuser to create this extension.
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          
          Caused by:
          PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
          HINT:  Must be superuser to create this extension.
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          Tasks: TOP => db:migrate
          (See full trace by running task with --trace)
    

    ================================================================================
    Error executing action run on resource ‘bash[migrate gitlab-rails database]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
    STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
    – column_exists?(:events, :fingerprint)
    -> 0.0030s
    – add_column(:events, :fingerprint, :binary)
    -> 0.0095s
    – transaction_open?()
    -> 0.0000s
    – execute(“ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n”)
    -> 0.0100s
    – execute(“ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;”)
    -> 0.0162s
    == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
    – transaction_open?()
    -> 0.0000s
    – index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
    -> 0.0113s
    – add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
    -> 0.0216s
    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s)

    == 20200515152649 EnableBtreeGistExtension: migrating =========================
    – execute(“CREATE EXTENSION IF NOT EXISTS btree_gist”)
    rake aborted!
    StandardError: An error has occurred, this and all later migrations canceled:

    PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
    HINT: Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

    Caused by:
    ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
    HINT: Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

    Caused by:
    PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
    HINT: Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    STDERR:
    ---- End output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
    Ran “bash” “/tmp/chef-script20200829-1833-1bujtkw” returned 1

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

    55: bash “migrate gitlab-rails database” do
    56: code <<-EOH
    57: set -e
    58: log_file="#{node[‘gitlab’][‘gitlab-rails’][‘log_directory’]}/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log" 59: umask 077 60: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}
    61: STATUS={PIPESTATUS[0]} 62: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} {log_file}
    63: echo $STATUS > #{db_migrate_status_file}
    64: exit $STATUS
    65: EOH
    66: environment env_variables unless env_variables.empty?
    67: notifies :run, “execute[clear the gitlab-rails cache]”, :immediately
    68: dependent_services.each do |svc|
    69: notifies :restart, svc, :immediately
    70: end
    71: not_if “(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)”
    72: only_if { node[‘gitlab’][‘gitlab-rails’][‘auto_migrate’] }
    73: end

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file’

    bash(“migrate gitlab-rails database”) do
    action [:run]
    default_guard_interpreter :default
    command nil
    backup 5
    interpreter “bash”
    declared_type :bash
    cookbook_name “gitlab”
    recipe_name “database_migrations”
    code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}\n STATUS={PIPESTATUS[0]}\n chown git:git {log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49\n exit $STATUS\n"
    domain nil
    user nil
    not_if “(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49 | grep -Fx 0)”
    only_if { #code block }
    end

    System Info:

    chef_version=15.12.22
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client

Recipe: gitlab::puma

  • runit_service[puma] action restart (up to date)
  • runit_service[puma] action restart (up to date)
    Recipe: gitlab::sidekiq-cluster
  • sidekiq_service[sidekiq] action restart
    Recipe:
    • service[sidekiq] action nothing (skipped due to action :nothing)
    • runit_service[sidekiq] action restart (up to date)
      (up to date)
      Recipe: gitlab::gitlab-rails
  • execute[clear the gitlab-rails cache] action run
    • execute /opt/gitlab/bin/gitlab-rake cache:clear
      Recipe: gitaly::enable
  • runit_service[gitaly] action hup
    • send hup to runit_service[gitaly]

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
– column_exists?(:events, :fingerprint)
-> 0.0030s
– add_column(:events, :fingerprint, :binary)
-> 0.0095s
– transaction_open?()
-> 0.0000s
– execute(“ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n”)
-> 0.0100s
– execute(“ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;”)
-> 0.0162s
== 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
– transaction_open?()
-> 0.0000s
– index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
-> 0.0113s
– add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
-> 0.0216s
== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s)

== 20200515152649 EnableBtreeGistExtension: migrating =========================
– execute(“CREATE EXTENSION IF NOT EXISTS btree_gist”)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
HINT: Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

Caused by:
ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
HINT: Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

Caused by:
PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
HINT: Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
Ran “bash” “/tmp/chef-script20200829-1833-1bujtkw” returned 1

Running handlers complete
Chef Infra Client failed. 13 resources updated in 01 minutes 06 seconds

Had the same issue and here’s the instruction:

Basically you’ll need to psql to your gitlab DB with a super user, and CREATE EXTENSION IF NOT EXISTS btree_gist;

PostgreSQL 13 claims to have btree_gist whitelisted as trusted extension, which can be created without admin permission.
So upgrade may also be an option (never tried, I’m using 12 because that’s the highest version available in CentOS’s SCL for now).