Updating gitlab-ce (docker version) database not working

How to Use GitLab Upgrade
1

I am running a docker Gitlab image via docker-compose. In my update procedure I will do a docker-compose pull and a docker-compose up -d afterwards.

This usually works quite fine, but my current update attempts fail with the following log entries (retrieved with docker logs -f gitlab).

I think I can figure out that postgresql is not running on the expected port which is why the db migration fails.

What do I have to do to fix this?

...
* postgresql_user[gitlab] action create
* execute[create gitlab postgresql user] action run (skipped due to not_if)
 (up to date)
* execute[create gitlabhq_production database] action run (skipped due to not_if)
* postgresql_user[gitlab_replicator] action create
* execute[create gitlab_replicator postgresql user] action run (skipped due to not_if)
* execute[set options for gitlab_replicator postgresql user] action run (skipped due to not_if)
 (up to date)
* postgresql_extension[pg_trgm] action enable
* postgresql_query[enable pg_trgm extension] action run (skipped due to only_if)
 (up to date)
* postgresql_extension[btree_gist] action enable
* postgresql_query[enable btree_gist extension] action run (skipped due to only_if)
 (up to date)
* ruby_block[warn pending postgresql restart] action run (skipped due to only_if)
* execute[reload postgresql] action nothing (skipped due to action :nothing)
* execute[start postgresql] action nothing (skipped due to action :nothing)
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] rake aborted!
          PG::ConnectionBad: could not connect to server: No such file or directory
            Is the server running locally and accepting
            connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          Tasks: TOP => gitlab:db:configure
          (See full trace by running task with --trace)

================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200707-25-popztg" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
  Is the server running locally and accepting
  connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  "/tmp/chef-script20200707-25-popztg" ----
Ran "bash"  "/tmp/chef-script20200707-25-popztg" returned 1

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 55: bash "migrate gitlab-rails database" do
 56:   code <<-EOH
 57:     set -e
 58:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
 59:     umask 077
 60:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
 61:     STATUS=${PIPESTATUS[0]}
 62:     chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
 63:     echo $STATUS > #{db_migrate_status_file}
 64:     exit $STATUS
 65:   EOH
 66:   environment env_variables unless env_variables.empty?
 67:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 68:   dependent_services.each do |svc|
 69:     notifies :restart, svc, :immediately
 70:   end
 71:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
 72:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
 73: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:55:in `from_file'

bash("migrate gitlab-rails database") do
  action [:run]
  default_guard_interpreter :default
  command nil
  backup 5
  interpreter "bash"
  declared_type :bash
  cookbook_name "gitlab"
  recipe_name "database_migrations"
  code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    chown git:git ${log_file}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f\n    exit $STATUS\n"
  domain nil
  user nil
  not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8411f47246f | grep -Fx 0)"
  only_if { #code block }
end

System Info:
------------
chef_version=14.14.29
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
* runit_service[gitaly] action hup
- send hup to runit_service[gitaly]
Recipe: package::sysctl
* execute[reload all sysctl conf] action run
[execute] * Applying /etc/sysctl.d/10-console-messages.conf ...
          sysctl: setting key "kernel.printk": Read-only file system
          * Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
          sysctl: setting key "net.ipv6.conf.all.use_tempaddr": Read-only file system
          sysctl: setting key "net.ipv6.conf.default.use_tempaddr": Read-only file system
          * Applying /etc/sysctl.d/10-kernel-hardening.conf ...
          sysctl: setting key "kernel.kptr_restrict": Read-only file system
          * Applying /etc/sysctl.d/10-link-restrictions.conf ...
          sysctl: setting key "fs.protected_hardlinks": Read-only file system
          sysctl: setting key "fs.protected_symlinks": Read-only file system
          * Applying /etc/sysctl.d/10-magic-sysrq.conf ...
          sysctl: setting key "kernel.sysrq": Read-only file system
          * Applying /etc/sysctl.d/10-network-security.conf ...
          sysctl: setting key "net.ipv4.conf.default.rp_filter": Read-only file system
          sysctl: setting key "net.ipv4.conf.all.rp_filter": Read-only file system
          * Applying /etc/sysctl.d/10-ptrace.conf ...
          sysctl: setting key "kernel.yama.ptrace_scope": Read-only file system
          * Applying /etc/sysctl.d/10-zeropage.conf ...
          * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf ...
          * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf ...
          * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf ...
          sysctl: setting key "vm.mmap_min_addr": Read-only file system
          sysctl: setting key "kernel.sem": Read-only file system
          sysctl: setting key "kernel.shmall": Read-only file system
          sysctl: setting key "kernel.shmmax": Read-only file system
          * Applying /etc/sysctl.d/99-sysctl.conf ...
          * Applying /etc/sysctl.conf ...
- execute sysctl -e --system

Running handlers:
Running handlers complete
Chef Client failed. 150 resources updated in 01 minutes 09 seconds
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200707-25-popztg" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  "/tmp/chef-script20200707-25-popztg" ----
Ran "bash"  "/tmp/chef-script20200707-25-popztg" returned 1
2

Uh boy, turns out that RTFM really does help sometimes… (see https://docs.gitlab.com/omnibus/update/gitlab_13_changes.html)

I was on Gitlab 12.9.4 – which was still using postgresql 10.12. My docker-compose.yml was using the latest tag for docker updates, which failed because the conversion from postgresql 10.12 to 11 took place in tag 12.10.14-ce.0.

So, what I basically did is update to that version prior to update to latest (i.e. version 13) again.

Hope this might help others, glad I kept on looking for a solution…

1 Like