Unable to upgrade to Postgres 12

How to Use GitLab Upgrade
1

I tried to upgrade to Postgresql 12. But the upgrade failed with permission denied error.

Now I am not even able to run gitlab-ctl pg-upgrade -v 12

root@git:/# gitlab-ctl pg-upgrade -V 12
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 12.4
Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade: OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details

Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: gitaly: 0s, normally up
ok: down: logrotate: 1s, normally up
ok: down: registry: 0s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: sshd: 1s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database:Error initializing database for 12.4
STDOUT: The files belonging to this database system will be owned by user "gitlab-psql".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/opt/gitlab/postgresql/data.12 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 20
selecting default shared_buffers ... 400kB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... 
STDERR: sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot open /dev/null: Permission denied
sh: 1: cannot create /dev/null: Permission denied
child process exited with exit code 2
initdb: removing contents of data directory "/var/opt/gitlab/postgresql/data.12"
== Fatal error ==
Please check the output and try again
== Reverting ==
ok: down: postgresql: 1s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 17733) 1s
== Reverted ==
== Reverted to 11.9. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: gitaly: (pid 17745) 1s
ok: run: logrotate: (pid 17765) 0s
ok: run: registry: (pid 17778) 1s
ok: run: sidekiq: (pid 17786) 0s
ok: run: sshd: (pid 17794) 0s
Toggling services: OK
root@git:/# echo "" > /dev/null
root@git:/# gitlab-ctl pg-upgrade -v 12
omnibus-ctl: version unknown
root@git:/#

Update

The omnibus-ctl: version unknown error is gone after restarting the container. But pg-upgrade is still failing with error

sh: 1: cannot open /dev/null: Permission denied

2

I was able to upgrade, after I recreated /dev/null

root@git:/# mv /dev/null{,.bak}
root@git:/# mknod /dev/null c 1 3
root@git:/# chmod 666 /dev/null

But when I checked after upgrade, the file permission had reverted to 0666 and ownership to git:root.

crwx------ 1 git root 1, 3 Dec 30 13:26 /dev/null

Should we run gitlab-ctl pg-upgrade as git and not root?

3

What container runtime are you using? Podman previously had some issues with overlapping mounts and /dev/null. If /dev/null ends up missing, the first process to write to it will probably end up owning it, which is likely going to continue to give you issues during restarts/upgrades.