Gitlab reconfigure Update crash of today

I updated gitlab today using apt udpate/upgrade and then ran gitlab-ctl reconfigure as root. Here is the error I got:

Recipe: gitlab::database_migrations

  • bash[migrate gitlab-rails database] action run
    [execute] == 20200504191813 AddFingerprintToEvents: migrating ===========================
    – column_exists?(:events, :fingerprint)
    -> 0.0030s
    – add_column(:events, :fingerprint, :binary)
    -> 0.0095s
    – transaction_open?()
    -> 0.0000s
    – execute(“ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n”)
    -> 0.0100s
    – execute(“ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;”)
    -> 0.0162s
    == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

          == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
          -- transaction_open?()
             -> 0.0000s
          -- index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
             -> 0.0113s
          -- add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
             -> 0.0216s
          == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s) 
          
          == 20200515152649 EnableBtreeGistExtension: migrating =========================
          -- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
          rake aborted!
          StandardError: An error has occurred, this and all later migrations canceled:
          
          PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
          HINT:  Must be superuser to create this extension.
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          
          Caused by:
          ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
          HINT:  Must be superuser to create this extension.
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          
          Caused by:
          PG::InsufficientPrivilege: ERROR:  permission denied to create extension "btree_gist"
          HINT:  Must be superuser to create this extension.
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          Tasks: TOP => db:migrate
          (See full trace by running task with --trace)
    

    ================================================================================
    Error executing action run on resource ‘bash[migrate gitlab-rails database]’

    Mixlib::ShellOut::ShellCommandFailed

    Expected process to exit with [0], but received ‘1’
    ---- Begin output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
    STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
    – column_exists?(:events, :fingerprint)
    -> 0.0030s
    – add_column(:events, :fingerprint, :binary)
    -> 0.0095s
    – transaction_open?()
    -> 0.0000s
    – execute(“ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n”)
    -> 0.0100s
    – execute(“ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;”)
    -> 0.0162s
    == 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
    – transaction_open?()
    -> 0.0000s
    – index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
    -> 0.0113s
    – add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
    -> 0.0216s
    == 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s)

    == 20200515152649 EnableBtreeGistExtension: migrating =========================
    – execute(“CREATE EXTENSION IF NOT EXISTS btree_gist”)
    rake aborted!
    StandardError: An error has occurred, this and all later migrations canceled:

    PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
    HINT: Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

    Caused by:
    ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
    HINT: Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

    Caused by:
    PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
    HINT: Must be superuser to create this extension.
    /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
    /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
    Tasks: TOP => db:migrate
    (See full trace by running task with --trace)
    STDERR:
    ---- End output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
    Ran “bash” “/tmp/chef-script20200829-1833-1bujtkw” returned 1

    Resource Declaration:

    In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

    55: bash “migrate gitlab-rails database” do
    56: code <<-EOH
    57: set -e
    58: log_file="#{node[‘gitlab’][‘gitlab-rails’][‘log_directory’]}/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log" 59: umask 077 60: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}
    61: STATUS={PIPESTATUS[0]} 62: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} {log_file}
    63: echo $STATUS > #{db_migrate_status_file}
    64: exit $STATUS
    65: EOH
    66: environment env_variables unless env_variables.empty?
    67: notifies :run, “execute[clear the gitlab-rails cache]”, :immediately
    68: dependent_services.each do |svc|
    69: notifies :restart, svc, :immediately
    70: end
    71: not_if “(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)”
    72: only_if { node[‘gitlab’][‘gitlab-rails’][‘auto_migrate’] }
    73: end

    Compiled Resource:

    Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file’

    bash(“migrate gitlab-rails database”) do
    action [:run]
    default_guard_interpreter :default
    command nil
    backup 5
    interpreter “bash”
    declared_type :bash
    cookbook_name “gitlab”
    recipe_name “database_migrations”
    code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}\n STATUS={PIPESTATUS[0]}\n chown git:git {log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49\n exit $STATUS\n"
    domain nil
    user nil
    not_if “(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-47e6e804d9202096c772d1e85ab70abb-f4a969f7f49 | grep -Fx 0)”
    only_if { #code block }
    end

    System Info:

    chef_version=15.12.22
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client

Recipe: gitlab::puma

  • runit_service[puma] action restart (up to date)
  • runit_service[puma] action restart (up to date)
    Recipe: gitlab::sidekiq-cluster
  • sidekiq_service[sidekiq] action restart
    Recipe:
    • service[sidekiq] action nothing (skipped due to action :nothing)
    • runit_service[sidekiq] action restart (up to date)
      (up to date)
      Recipe: gitlab::gitlab-rails
  • execute[clear the gitlab-rails cache] action run
    • execute /opt/gitlab/bin/gitlab-rake cache:clear
      Recipe: gitaly::enable
  • runit_service[gitaly] action hup
    • send hup to runit_service[gitaly]

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
STDOUT: == 20200504191813 AddFingerprintToEvents: migrating ===========================
– column_exists?(:events, :fingerprint)
-> 0.0030s
– add_column(:events, :fingerprint, :binary)
-> 0.0095s
– transaction_open?()
-> 0.0000s
– execute(“ALTER TABLE events\nADD CONSTRAINT check_97e06e05ad\nCHECK ( octet_length(fingerprint) <= 128 )\nNOT VALID;\n”)
-> 0.0100s
– execute(“ALTER TABLE events VALIDATE CONSTRAINT check_97e06e05ad;”)
-> 0.0162s
== 20200504191813 AddFingerprintToEvents: migrated (0.0617s) ==================

== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrating =======
– transaction_open?()
-> 0.0000s
– index_exists?(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
-> 0.0113s
– add_index(:events, [:target_type, :target_id, :fingerprint], {:using=>:btree, :unique=>true, :algorithm=>:concurrently})
-> 0.0216s
== 20200504200709 AddIndexOnFingerprintAndTargetTypeToEvents: migrated (0.0335s)

== 20200515152649 EnableBtreeGistExtension: migrating =========================
– execute(“CREATE EXTENSION IF NOT EXISTS btree_gist”)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
HINT: Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

Caused by:
ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
HINT: Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in

Caused by:
PG::InsufficientPrivilege: ERROR: permission denied to create extension “btree_gist”
HINT: Must be superuser to create this extension.
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200515152649_enable_btree_gist_extension.rb:7:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:54:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” “/tmp/chef-script20200829-1833-1bujtkw” ----
Ran “bash” “/tmp/chef-script20200829-1833-1bujtkw” returned 1

Running handlers complete
Chef Infra Client failed. 13 resources updated in 01 minutes 06 seconds