Pg_dump server version mismatch

I am running GitLab CE 15.5.4 in a Docker with external PostgreSQL 14.6. An attempt to take backup of GitLab leads to “server version mismatch” error thrown by pg_dump as shown below.

$ sudo docker exec -t gitlab gitlab-backup create
2023-01-23 17:41:37 +0000 -- Dumping main_database ... 
Dumping PostgreSQL database gitlab ... pg_dump: error: server version: 14.6 (Ubuntu 14.6-0ubuntu0.22.04.1); pg_dump version: 13.6
pg_dump: error: aborting because of server version mismatch

When I looked for PostgreSQL packages inside Docker, I find none.

root@gitlab:/# apt list --installed | grep postgres

Though PostgreSQL 12 and 13 are available in Docker.

root@gitlab:/# ls -l /opt/gitlab/embedded/postgresql/
total 8
drwxr-xr-x 5 root root 4096 Nov 11 20:36 12
drwxr-xr-x 5 root root 4096 Nov 11 20:36 13

What is the recommended way to install and use a different version of pg_dump when running GitLab CE inside Docker?


Looking at PostgreSQL versions shipped with Omnibus GitLab | GitLab it seems that PostgreSQL 14.6 has not been added to the list of supported versions yet.

Not sure why it worked with provisioning initially during setup - was the database server upgraded to PostgreSQL 14.6 recently, and the initial PostgreSQL version matched what GitLab supports before?

Googled for gitlab docker pg_dump upgrade which brought up a potential workaround to install a newer pg_dump package into the running container to perform the database. Postgres version missmatch (#215746) · Issues · / GitLab · GitLab

This workaround is likely needed until there is official support for PostgreSQL 14.x in GitLab, issue to track: Research spike: Adding support for PG14 (#7396) · Issues · / omnibus-gitlab · GitLab Once 14.x is supported, I’d suggest “pinning” the server version to what GitLab supports.

This is a new installation of GitLab in Docker. I configured it to use an external PostgreSQL (14.6) as described here - Configure GitLab using an external PostgreSQL service.

GitLab works pretty well with PostgreSQL 14.6. Only backup fails due to version mismatch, which is expected.

I also run a Discourse instance in similar way and there also backup fails due to same reason. To fix the backup issue for Discourse, I will just uninstall postgres packages from Docker and install newer version.

# remove postgres packages
apt remove postgresql-13 postgresql-client-13 postgresql-client-common postgresql-common

# add PostgreSQL repository
sudo sh -c 'echo "deb $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - | sudo apt-key add -
apt update

# install same version of Postgresql client as we have for server
apt install -y postgresql-client-14

Though for GitLab there are no PostgreSQL packages installed in Docker hence above-mentioned trick to fix backup issue does not work.

Database settings | GitLab outlines a way to have gitlab use different versions of the psql tools. The main step there is that once you have the tools installed, you need to symlink them in /opt/gitlab/bin/. Our own psql tools are symlinked in /opt/gitlab/embedded/bin but ‘/opt/gitlab/bin’ we have first in the path for our services, so if you symlink there, your versions of the tools will be used, and you don’t have to worry about us overwriting your symlink each update.

This should allow you to use a similar workaround to your discourse solution. You just need to skip the remove step. And add a symlink step at the end.

1 Like

Yes, I saw that document and tried tin install PostgreSQL 14 client on the Docker instance where GitLab is running. But GitLab’s Docker image seems to be outdated from PostgreSQL 14’s perspective.

sh -c 'echo "deb jammy-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

wget --quiet -O - | apt-key add -

apt install postgresql-client-14

root@gitlab:/# apt install postgresql-client-14
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 postgresql-client-14 : Depends: libpq5 (>= 14.6) but it is not going to be installed
                        Depends: libc6 (>= 2.34) but 2.31-0ubuntu9.9 is to be installed
                        Depends: libssl3 (>= 3.0.0~~alpha1) but it is not installable
E: Unable to correct problems, you have held broken packages.

This error looks like a package dependency mismatch between distribution and to-be-installed package.

is the codename for Ubuntu 22.04 LTS. The GitLab docker image uses Ubuntu 20.04 Focal as base. docker/Dockerfile · master · / omnibus-gitlab · GitLab

Postgresql has focal packages available in the apt repository. PostgreSQL: Linux downloads (Ubuntu)

1 Like

Thank You @dnsmichi for hint about jammy vs focal. Actually on GitLab Docker the lsb_release command was not available. So I ran it on host machine and took the output value jammy while adding PostgreSQL repositories. That was causing broken packages error.

By using focal, I am able to install PostgreSQL 14.6 client on GitLab Docker and now backup is also working perfectly fine.

Thanks for all the help that you and @twk3 extended. Much appreciated :pray:t5:

Glad you could solve your problem :slight_smile:

I remember this problem when we tried to implement a “print OS version from the binary” feature in a previous OSS project I maintained. lsb_release needs to be installed via apt -y install lsb-release. An alternative method is cat /etc/os-release which has become a standard in most recent Linux distributions (some older use /etc/redhat-version, /etc/SuSE-version, etc.).

In this case, I did not have access to a running container image on my iPad. My path was to search for the Docker image sources (gitlab docker image), found the image on Docker Hub , and the links to the Dockerfile. If there were no links, the search would have been a bit longer - I then usually inspect the organization of projects on GitHub/GitLab and search for Dockerfile (or FROM including the whitespace to match Docker syntax in the content). Hope explaining this debugging path helps :slight_smile:

1 Like