I am running GitLab CE 15.5.4 in a Docker with external PostgreSQL 14.6. An attempt to take backup of GitLab leads to “server version mismatch” error thrown by pg_dump as shown below.
$ sudo docker exec -t gitlab gitlab-backup create
2023-01-23 17:41:37 +0000 -- Dumping main_database ...
Dumping PostgreSQL database gitlab ... pg_dump: error: server version: 14.6 (Ubuntu 14.6-0ubuntu0.22.04.1); pg_dump version: 13.6
pg_dump: error: aborting because of server version mismatch
When I looked for PostgreSQL packages inside Docker, I find none.
root@gitlab:/# apt list --installed | grep postgres
Though PostgreSQL 12 and 13 are available in Docker.
root@gitlab:/# ls -l /opt/gitlab/embedded/postgresql/
total 8
drwxr-xr-x 5 root root 4096 Nov 11 20:36 12
drwxr-xr-x 5 root root 4096 Nov 11 20:36 13
What is the recommended way to install and use a different version of pg_dump when running GitLab CE inside Docker?
Not sure why it worked with provisioning initially during setup - was the database server upgraded to PostgreSQL 14.6 recently, and the initial PostgreSQL version matched what GitLab supports before?
GitLab works pretty well with PostgreSQL 14.6. Only backup fails due to version mismatch, which is expected.
I also run a Discourse instance in similar way and there also backup fails due to same reason. To fix the backup issue for Discourse, I will just uninstall postgres packages from Docker and install newer version.
# remove postgres packages
apt remove postgresql-13 postgresql-client-13 postgresql-client-common postgresql-common
# add PostgreSQL repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
apt update
# install same version of Postgresql client as we have for server
apt install -y postgresql-client-14
Though for GitLab there are no PostgreSQL packages installed in Docker hence above-mentioned trick to fix backup issue does not work.
Database settings | GitLab outlines a way to have gitlab use different versions of the psql tools. The main step there is that once you have the tools installed, you need to symlink them in /opt/gitlab/bin/. Our own psql tools are symlinked in /opt/gitlab/embedded/bin but ‘/opt/gitlab/bin’ we have first in the path for our services, so if you symlink there, your versions of the tools will be used, and you don’t have to worry about us overwriting your symlink each update.
This should allow you to use a similar workaround to your discourse solution. You just need to skip the remove step. And add a symlink step at the end.
Yes, I saw that document and tried tin install PostgreSQL 14 client on the Docker instance where GitLab is running. But GitLab’s Docker image seems to be outdated from PostgreSQL 14’s perspective.
sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt jammy-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
apt install postgresql-client-14
root@gitlab:/# apt install postgresql-client-14
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
postgresql-client-14 : Depends: libpq5 (>= 14.6) but it is not going to be installed
Depends: libc6 (>= 2.34) but 2.31-0ubuntu9.9 is to be installed
Depends: libssl3 (>= 3.0.0~~alpha1) but it is not installable
E: Unable to correct problems, you have held broken packages.
Thank You @dnsmichi for hint about jammy vs focal. Actually on GitLab Docker the lsb_release command was not available. So I ran it on host machine and took the output value jammy while adding PostgreSQL repositories. That was causing broken packages error.
By using focal, I am able to install PostgreSQL 14.6 client on GitLab Docker and now backup is also working perfectly fine.
Thanks for all the help that you and @twk3 extended. Much appreciated
I remember this problem when we tried to implement a “print OS version from the binary” feature in a previous OSS project I maintained. lsb_release needs to be installed via apt -y install lsb-release. An alternative method is cat /etc/os-release which has become a standard in most recent Linux distributions (some older use /etc/redhat-version, /etc/SuSE-version, etc.).
In this case, I did not have access to a running container image on my iPad. My path was to search for the Docker image sources (gitlab docker image), found the image on Docker Hub , and the links to the Dockerfile. If there were no links, the search would have been a bit longer - I then usually inspect the organization of projects on GitHub/GitLab and search for Dockerfile (or FROM including the whitespace to match Docker syntax in the content). Hope explaining this debugging path helps
