Can't upgrade gitlab from 12.6.4 to 13.5.1

winnertako · October 28, 2020, 6:17pm

Hello

I want to upgrade my gitlab-ce 12.6.4 and I’m facing some problems. I know I have to got from 12.6.4 -> 12.10.14 -> 13.0.14 -> 13.5.1 but I’ve one problem.

When I’m trying to upgrade from the 12.6.4 to 12.10.14 I’ve problems with the postgresql upgrade.

This is my current version:

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	3.2.12
Git Version:	2.24.1
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.6.4
Revision:	70900054dfe
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.6.14
URL:		https://xtpo.com
HTTP Clone URL:	https://xtpo.com/some-group/some-project.git
SSH Clone URL:	git@xtpo.com:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	10.3.0
Repository storage paths:
- default: 	/mnt/gitlab-data0/repositories
- gitlab-data1: 	/mnt/gitlab-data1/repositories
- gitlab-data2: 	/mnt/gitlab-data2/repositories
- gitlab-data3: 	/mnt/gitlab-data3/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

If I try to go from the 12.6.4 -> 12.10.14 the upgrade fails because the postgres.

yum install gitlab-ce-12.10.14-ce.0.el7.x86_64

gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
2020-10-28 17:20:04 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2020-10-28 17:36:26 +0000 -- done
2020-10-28 17:36:26 +0000 -- Dumping repositories ...
2020-10-28 17:36:26 +0000 -- [SKIPPED]
2020-10-28 17:36:26 +0000 -- Dumping uploads ...
2020-10-28 17:36:26 +0000 -- [SKIPPED]
2020-10-28 17:36:26 +0000 -- Dumping builds ...
2020-10-28 17:36:26 +0000 -- [SKIPPED]
2020-10-28 17:36:26 +0000 -- Dumping artifacts ...
2020-10-28 17:36:26 +0000 -- [SKIPPED]
2020-10-28 17:36:26 +0000 -- Dumping pages ...
2020-10-28 17:36:26 +0000 -- [SKIPPED]
2020-10-28 17:36:26 +0000 -- Dumping lfs objects ...
2020-10-28 17:36:26 +0000 -- [SKIPPED]
2020-10-28 17:36:26 +0000 -- Dumping container registry images ...
2020-10-28 17:36:26 +0000 -- [DISABLED]
Creating backup archive: 1603906586_2020_10_28_12.6.4_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... done. (3 removed)
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.
gitlab preinstall: Automatically backing up /etc/gitlab
Running configuration backup
Creating configuration backup archive: gitlab_config_1603906737_2020_10_28.tar
/etc/gitlab/
/etc/gitlab/gitlab-secrets.json
/etc/gitlab/trusted-certs/
/etc/gitlab/ssl/
/etc/gitlab/ssl/xtpo_com.crt
/etc/gitlab/ssl/xtpo_com.key
/etc/gitlab/ssl/xtop.com.crt
/etc/gitlab/ssl/xtpo.com.key
/etc/gitlab/gitlab.rb
Configuration backup archive complete: /etc/gitlab/config_backup/gitlab_config_1603906737_2020_10_28.tar
  Updating   : gitlab-ce-12.10.14-ce.0.el7.x86_64                                                                                                                                                       1/2
  Cleanup    : gitlab-ce-12.6.4-ce.0.el7.x86_64                                                                                                                                                         2/2
Checking PostgreSQL executables:Starting Chef Client, version 14.14.29
resolving cookbooks for run list: ["gitlab::config", "postgresql::bin"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - postgresql (0.1.0)
  - package (0.1.0)
  - redis (0.1.0)
  - registry (0.1.0)
  - monitoring (0.1.0)
  - mattermost (0.1.0)
  - consul (0.1.0)
  - gitaly (0.1.0)
  - nginx (0.1.0)
  - letsencrypt (0.1.0)
  - runit (4.3.0)
  - praefect (0.1.0)
  - crond (0.1.0)
  - acme (4.1.1)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 3 resources
Recipe: postgresql::bin
  * ruby_block[check_postgresql_version] action run (skipped due to not_if)
  * ruby_block[check_postgresql_version_is_deprecated] action run (skipped due to not_if)
  * ruby_block[Link postgresql bin files to the correct version] action run
    - execute the ruby block Link postgresql bin files to the correct version

Running handlers:
Running handlers complete
Chef Client finished, 1/3 resources updated in 20 seconds
Checking PostgreSQL executables: OK
Checking if Grafana needs to be reset:
Status file found. Grafana is not vulnerable. Skipping reset.
Checking if Grafana needs to be reset: OK
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 11.7
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 0s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: grafana: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: node-exporter: 0s, normally up
ok: down: postgres-exporter: 1s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: sidekiq: 1s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data:Error upgrading the data to version 11.7
STDOUT:
STDERR: timed out
Upgrading the data: NOT OK
== Fatal error ==
Error running pg_upgrade, please check logs
== Reverting ==
ok: down: postgresql: 627s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 10548) 0s
== Reverted ==
== Reverted to 9.6.17. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 10562) 0s
ok: run: gitaly: (pid 10564) 1s
ok: run: gitlab-exporter: (pid 10567) 0s
ok: run: grafana: (pid 10589) 1s
ok: run: logrotate: (pid 10599) 0s
ok: run: node-exporter: (pid 10612) 1s
ok: run: postgres-exporter: (pid 10619) 0s
ok: run: prometheus: (pid 10638) 0s
ok: run: redis-exporter: (pid 10650) 0s
ok: run: sidekiq: (pid 10652) 0s
Toggling services: OK
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
warning: %posttrans(gitlab-ce-12.10.14-ce.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-12.10.14-ce.0.el7.x86_64
  Verifying  : gitlab-ce-12.10.14-ce.0.el7.x86_64                                                                                                                                                       1/2
  Verifying  : gitlab-ce-12.6.4-ce.0.el7.x86_64                                                                                                                                                         2/2

Updated:
  gitlab-ce.x86_64 0:12.10.14-ce.0.el7

Complete!

The GitLab are updated to the target version, but the postgresql not.

If I disable the automatic PostgreSQL upgrade during GitLab package upgrade I can go to the 12.10.14. After that I’ve to upgrade my postgres before the GitLab 13.0.14 version but when I’m trying to upgrade manually I’ve the same issue.

Any idea what I’m doing wrong? If i go to /var/opt/gitlab/postgresql/pg_upgrade_server.log doesn’t show up any error…

Thank you.

iwalker · October 28, 2020, 6:59pm

Hi,

From what I remember you should have already been on postgres 10 before going to 12.10.14 Can you try running:

gitlab-ctl pg-upgrade

and then try to upgrade to 12.10.14 after this. I know when I was on 12.9.3 (I think) it was postgres 10.

grove · October 29, 2020, 9:11am

Postgres 10 didn’t become a requirement until GitLab 13, so what you’re trying is fine.

Unless disabled the postgresql upgrade was attempted a few versions before it became a requirement, and that must be what @iwalker remember seeing the effects of.

I would also like to see the output of the gitlab-ctl pg-upgrade that @iwalker gave.

.Henrik

winnertako · October 29, 2020, 10:54am

Hello @iwalker, @grove ,

I’ve the same error when I tried to do the gitlab-ctl pg-upgrade before the upgrade to the 12.10.4:

[root@gitlab-test ~]# gitlab-ctl pg-upgrade
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 11.7
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details

Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 0s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 1s, normally up
ok: down: grafana: 0s, normally up
ok: down: logrotate: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 1s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: sidekiq: 0s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data:

Error upgrading the data to version 11.7
STDOUT:
STDERR: timed out
Upgrading the data: NOT OK
== Fatal error ==
Error running pg_upgrade, please check logs
== Reverting ==
ok: down: postgresql: 636s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 16767) 1s
== Reverted ==
== Reverted to 9.6.17. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 16785) 0s
ok: run: gitaly: (pid 16787) 1s
ok: run: gitlab-exporter: (pid 16790) 0s
ok: run: grafana: (pid 16797) 0s
ok: run: logrotate: (pid 16817) 1s
ok: run: node-exporter: (pid 16819) 0s
ok: run: postgres-exporter: (pid 16826) 1s
ok: run: prometheus: (pid 16830) 0s
ok: run: redis-exporter: (pid 16838) 1s
ok: run: sidekiq: (pid 16855) 0s
Toggling services: OK

Thank you.

iwalker · October 29, 2020, 11:24am

The problem here is, a version of gitlab has been installed higher than it’s actually supporting. The database should have been on 10.x before installing the version that is forcing upgrade to 11.x

12.10.x is attempting to do upgrade to 11.7, but the host is on 9.x so it’s going to fail. 12.0 and higher should have upgraded the 9.x release to 10.x but it seems for some reason that hasn’t happened. Maybe it was disabled, or maybe the upgrade path wasn’t followed correctly to ensure that the upgrade went smoothly, including the postgres upgrades. But impossible to tell without all details.

I think the only way around this is to restore your server to it’s previous version (I presume you have a backup??), and ensure it’s on postgres 10.7 before going any further, and update to 12.10.x once this condition is met, so that it can be upgraded to postgres 11.x in gitlab 12.10.x

In gitlab 13.x postgres 11 only supported (12 is available but optional will be forced in gitlab 14.x), so the 12.x releases had to be on postgres 10 prior to upgrade.

winnertako · October 29, 2020, 12:48pm

Hi @iwalker,

Yes, I’ve one backup and I will try again the gitlab-ctl pg-upgrade.

I think I’ve already tested before and the result is the same.

I will give you feedback after this test.

Thank you.

grove · October 29, 2020, 1:36pm

Oh yeah, I was confused and was one major version behind on Postgres-versions (in what I wrote). @iwalker is right and probably much better to listen to than me.

iwalker · October 29, 2020, 1:39pm

@winnertako OK.

Once you restore it, would be nice to know what the Gitlab version is before we attempt upgrading any further to try and find out the upgrade path.

winnertako · October 29, 2020, 2:35pm

@iwalker thank you. I’ve the backup restored:

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	3.2.12
Git Version:	2.24.1
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.6.4
Revision:	70900054dfe
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.6.14
URL:		https://git-test.xtpo.com
HTTP Clone URL:	https://git-test.xtpo.com/some-group/some-project.git
SSH Clone URL:	git@git-test.xtpo.com:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	10.3.0
Repository storage paths:
- default: 	/mnt/gitlab-data0/repositories
- gitlab-data1: 	/mnt/gitlab-data1/repositories
- gitlab-data2: 	/mnt/gitlab-data2/repositories
- gitlab-data3: 	/mnt/gitlab-data3/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Now, I’ve to try the gitlab-ctl pg-upgrade right?

Thank you.

iwalker · October 29, 2020, 2:42pm

Yep try it now pg-upgrade.

Oh, and just check make sure this file doesn’t exist:

/etc/gitlab/disable-postgresql-upgrade

here explains about it checking for enough disk space also to complete upgrade, so providing there is enough free disk space it should run.

winnertako · October 29, 2020, 3:42pm

Hi @iwalker, @grove

The /gitlab/postgresql/data are placed in one mount point with 152G available:

[root@gitlab-test var]# sudo du -sh /var/opt/gitlab/postgresql/data
16G     /var/opt/gitlab/postgresql/data

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg1       200G   49G  152G  25% /mnt/gitlab-root0ls

I’ve tried the gitlab-ctl pg-upgrade but without success:

[root@gitlab-test ~]# gitlab-ctl pg-upgrade
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 10.9
Checking if we already upgraded: NOT OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Checking if postgresql['version'] is set: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details

Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 0s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: grafana: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: sidekiq: 0s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data:Error upgrading the data to version 10.9
STDOUT:
STDERR: timed out
Upgrading the data: NOT OK
Traceback (most recent call last):
        7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
        6: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
        5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
        4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
        3: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
        2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:175:in `block in load_file'
        1: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:228:in `general_upgrade'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `run_pg_upgrade': undefined method `ExecutionError' for GitlabCtl::Errors:Class (NoMethodError)
[root@gitlab-test ~]#

I can’t find anything, I’ve checked the /var/opt/gitlab/postgresql/pg_upgrade_server.log but I can’t find anything:

-----------------------------------------------------------------
  pg_upgrade run on Thu Oct 29 15:17:10 2020
-----------------------------------------------------------------

command: "/opt/gitlab/embedded/postgresql/9.6/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/mnt/gitlab-root0/gitlab/postgresql/data" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/mnt/gitlab-root0/gitlab/postgresql'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....LOG:  database system was shut down at 2020-10-29 15:16:58 GMT
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
 done
server started


command: "/opt/gitlab/embedded/postgresql/9.6/bin/pg_ctl" -w -D "/mnt/gitlab-root0/gitlab/postgresql/data" -o "" -m smart stop >> "pg_upgrade_server.log" 2>&1
waiting for server to shut down...LOG:  received smart shutdown request
.LOG:  shutting down
LOG:  database system is shut down
 done
server stopped


command: "/opt/gitlab/embedded/postgresql/10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/mnt/gitlab-root0/gitlab/postgresql/data.10" -o "-p 50432 -b -c synchronous_commit=off -c fsync=off -c full_page_writes=off  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/mnt/gitlab-root0/gitlab/postgresql'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2020-10-29 15:17:28.800 WET [10575] LOG:  listening on Unix socket "/mnt/gitlab-root0/gitlab/postgresql/.s.PGSQL.50432"
2020-10-29 15:17:28.815 WET [10576] LOG:  database system was shut down at 2020-10-29 15:17:08 WET
2020-10-29 15:17:28.818 WET [10575] LOG:  database system is ready to accept connections
 done
server started


command: "/opt/gitlab/embedded/postgresql/10/bin/pg_ctl" -w -D "/mnt/gitlab-root0/gitlab/postgresql/data.10" -o "" -m smart stop >> "pg_upgrade_server.log" 2>&1
waiting for server to shut down....2020-10-29 15:17:29.636 WET [10575] LOG:  received smart shutdown request
2020-10-29 15:17:29.637 WET [10575] LOG:  worker process: logical replication launcher (PID 10581) exited with exit code 1
2020-10-29 15:17:29.637 WET [10577] LOG:  shutting down
2020-10-29 15:17:29.649 WET [10575] LOG:  database system is shut down
 done
server stopped


command: "/opt/gitlab/embedded/postgresql/10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/mnt/gitlab-root0/gitlab/postgresql/data.10" -o "-p 50432 -b -c synchronous_commit=off -c fsync=off -c full_page_writes=off  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/mnt/gitlab-root0/gitlab/postgresql'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....2020-10-29 15:17:34.426 WET [10632] LOG:  listening on Unix socket "/mnt/gitlab-root0/gitlab/postgresql/.s.PGSQL.50432"
2020-10-29 15:17:34.446 WET [10633] LOG:  database system was shut down at 2020-10-29 15:17:34 WET
2020-10-29 15:17:34.449 WET [10632] LOG:  database system is ready to accept connections
 done
server started


command: "/opt/gitlab/embedded/postgresql/10/bin/pg_ctl" -w -D "/mnt/gitlab-root0/gitlab/postgresql/data.10" -o "" -m smart stop >> "pg_upgrade_server.log" 2>&1
waiting for server to shut down...2020-10-29 15:17:39.102 WET [10632] LOG:  received smart shutdown request
.2020-10-29 15:17:39.103 WET [10632] LOG:  worker process: logical replication launcher (PID 10638) exited with exit code 1
2020-10-29 15:17:39.103 WET [10634] LOG:  shutting down
2020-10-29 15:17:39.140 WET [10632] LOG:  database system is shut down
 done
server stopped

Thank you for your patience.

winnertako · October 30, 2020, 10:56am

Hello @grove, @iwalker

I think I found the solution. The database isn’t too large but it is necessary to increase the default timeout.

I’m trying with this option:

gitlab-ctl pg-upgrade --timeout 60000

It seems to work correctly:

Running handlers:
Running handlers complete
Chef Client finished, 5/743 resources updated in 57 seconds
Running reconfigure: OK
Waiting for Database to be running.
Database upgrade is complete, running analyze_new_cluster.sh

I will give you some feedback after this test.

Thank you

winnertako · November 2, 2020, 10:30am

Hi @iwalker, @grove

I’ve done the gitlab-ctl pg-upgrade with success but during the process. During pg-upgrade I got another erro STDERR: vacuumdb: vacuuming of database "gitlabhq_production" failed: ERROR: canceling statement due to statement timeout

Tried to work around by running "/opt/gitlab/embedded/bin/vacuumdb" --echo --all --analyze-in-stages -h /var/opt/gitlab/postgresql however that still failed with same error. Eventually adding -j 2 seemed to help: sudo su - gitlab-psql -c "/opt/gitlab/embedded/bin/vacuumdb -j 2 --all --analyze-in-stages -h /var/opt/gitlab/postgresql" .

sudo gitlab-rake gitlab:env:info

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	3.2.12
Git Version:	2.24.1
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.6.4
Revision:	70900054dfe
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.9
URL:		
HTTP Clone URL:	
SSH Clone URL:	
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	10.3.0
Repository storage paths:
- default: 	/mnt/gitlab-data0/repositories
- gitlab-data1: 	/mnt/gitlab-data1/repositories
- gitlab-data2: 	/mnt/gitlab-data2/repositories
- gitlab-data3: 	/mnt/gitlab-data3/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Now I will try to upgrade to Giltab `12.10.14. I will give you some feedback after this test.

Thank you

iwalker · November 2, 2020, 10:43am

@winnertako great info, this will help if anyone else has similar issues then can use the same parameters or slight modification of parameters to pass if it also fails for them.

Hopefully the next stages will go well, although unsure if they will need the additional parameters or not but I expect only if a db upgrade is required. Later it will go from postgres 10 to postgres 11 and you will be OK at least until gitlab 14.x when will be upgraded to postgres 12.

winnertako · November 2, 2020, 11:53am

Thank you @iwalker.

During the upgrade for Gitlab 12.10.14 the PostgreSQL doesn’t upgrade automatically. I had to done the upgrade manually (11.7 -> 11.9).

Now with the Gitlab 12.10.14 and PostgreSQL 11.9, can I go to the 13.5.1 or I have to go between versions? The PostgreSQL will be upgraded automatically?

Thank you.

iwalker · November 2, 2020, 12:07pm

You have to go to 13.0.0 first before you go to 13.5.1. I would suggest you do this:

12.10.14 --> 13.0.0
13.0.0 --> 13.1.0
13.1.0 --> 13.2.0
13.2.0 --> 13.3.0
13.3.0 --> 13.4.0
13.4.0 --> 13.5.0

and then do a standard upgrade using apt/yum or whatever to go to 13.5.1. When I did it, I did every single release between 12.9.3 and 13.5.x which was over 40 upgrades which probably wasn’t necessary but I prefer safe than sorry.

Because of migrations, you need to go to 13.0.0 first, and you will need to run the command to check if migrations have finished before upgrading again. You can do that with:

gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

make sure the result is 0 before continuing.

winnertako · November 2, 2020, 2:47pm

@iwalker thank your for your advice.

How can I run the suggest command?

I’ve tried but without success:

[root@gitlab ssl]# gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
Please specify a valid ruby command or the path of a script to run.
Run 'bin/rails runner -h' for help.

undefined method `remaining' for Gitlab::BackgroundMigration:Module

Thank you.

iwalker · November 2, 2020, 2:55pm

I just run mine as root, I have gitlab-ce version installed. Something doesn’t seem right with your installation if that’s the case as it’s not finding rails.

explains the commands and for which versions, including source version. Assuming yours is gitlab-ce and not a source install, then it should work.

winnertako · November 2, 2020, 3:25pm

Hi @iwalker,

You are right, something doesn’t seem right but I don’t know what is it. My PostgreSQL upgrades have to be done manually because if I try with the Gitlab package it will fail.

I will do everything again, step by step and I will give you some feedback after that.

Thank you for your help.

winnertako · November 5, 2020, 3:57pm

Hi @iwalker,

Finally I’m in the latest version. I’ve to jump from 12.6.4 -> 12.10.14 -> 13.014 -> 13.3.8 -> 13.5.3 and I’ve done all of my PostgreSQL upgrade manually.

But I’ve one last problem, to upgrade to this version I’ve to disable the selinux otherwise the gitlab-ctl reconfigure fails:

I can’t figure out why this is happening…

Thank you.