Upgrade from 16.6.6 to 16.6.7 ruby/glibc issue

I tried an upgrade from 16.6.6 to 16.6.7 but I get an error during upgrade:

The following packages will be upgraded:
  gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 1,088 MB of archives.
After this operation, 58.3 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic/main amd64 gitlab-ce amd64 16.7.7-ce.0 [1,088 MB]
Fetched 1,088 MB in 18s (59.8 MB/s)                                                                                                                                                                                                                                                                                           
(Reading database ... 184997 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_16.7.7-ce.0_amd64.deb ...
/opt/gitlab/embedded/bin/ruby: /lib/x86_64-linux-gnu/libcrypt.so.1: version `XCRYPT_2.0' not found (required by /opt/gitlab/embedded/lib/libruby.so.3.1)
/opt/gitlab/embedded/bin/ruby: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/libruby.so.3.1)
/opt/gitlab/embedded/bin/ruby: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /opt/gitlab/embedded/lib/libruby.so.3.1)
/opt/gitlab/embedded/bin/ruby: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /opt/gitlab/embedded/lib/libjemalloc.so.2)
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_16.7.7-ce.0_amd64.deb (--unpack):
 new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ce_16.7.7-ce.0_amd64.deb

I’m using Ubuntu 18.04.6 LTS with the Gitlab omnibus package.

If anyone has any ideas I’m open to suggestions, thanks in advance!

Sorry I don’t this this will help, but, I can’t even install 16.7.7. I am on an Ubuntu Jammy system and when I try

sudo apt install gitlab-ce=16.7.7-ce.0

It says:

Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Package gitlab-ce is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Version ‘16.7.7-ce.0’ for ‘gitlab-ce’ was not found

I’ve found this issue Ubuntu packages are ambiguously named and are missing libc version dependencies (#5991) · Issues · GitLab.org / omnibus-gitlab · GitLab which says it is possible to install a manual package through download and not apt, and use the wrong distribution. Later attempts to upgrade the package can fail with wrong glibc dependencies.

I need to guess here - was LD_LIBRARY_PATH modified on the system? I found https://www.linuxquestions.org/questions/linux-software-2/[ruby]-xcrypt_2-0-not-found-4175669253/ after searching for libcrypt.so.1: version XCRYPT_2.0’ not found`

What else is running on this Linux host, how often is it upgraded, is it a VM or bare metal?

Ubuntu Bionic was EOL in May 2023. Maybe an alternative route can be to upgrade to Ubuntu 20 or 22 LTS, and then continue with the upgrade path for GitLab.

Cannot reproduce this in a fresh Docker container.

docker run -ti ubuntu:jammy sh

apt update
apt -y install curl 

curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | bash

apt update 

apt install gitlab-ce=16.7.7-ce.0

Maybe apt cache was not refreshed yet, or the GitLab package repository is not setup properly? apt update should help.

This is installed on a bare metal system. I have been upgrading it successfully for the last year at each security release without issue. I am currently at 16.7.6, and trying to install 16.7.7 over it.

When I try to do sudo apt update I get the following:

Hit:1 Index of /ubuntu jammy InRelease
Hit:2 Index of /ubuntu jammy-security InRelease
Hit:3 Index of /ubuntu jammy-updates InRelease
Hit:5 Index of /ubuntu jammy-backports InRelease
Hit:6 Index of /ondrej/apache2/ubuntu jammy InRelease
Get:4 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu focal InRelease [23.3 kB]
Err:4 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu focal InRelease
The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) packages@gitlab.com
Fetched 23.3 kB in 1s (18.8 kB/s)
Reading package lists… Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) packages@gitlab.com
W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/dists/focal/InRelease The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) packages@gitlab.com
W: Some index files failed to download. They have been ignored, or old ones used instead.

One other note… I have put gitlab-ce on hold and am upgrading it manually following the online upgrade paths so that I am not on the very latest version. I try to keep about one or two versions back but keep up to date on the security releases.

Here is my ubuntu version:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy

This error prevents updating the apt cache, thus leading to no avail of the package.

I’ve seen other users discussing GPG expiry problems on various distributions in Gitlab repo GPG key no longer valid + Gitlab Inc. instructions on replacement don't seem to work on Rocky 8 Suggest reviewing whether one of the comments helps resolve the issue.

This worked for me! Thank you!

1 Like

Yes, I think I made the mistake of downloading the incorrect deb off of packages.gitlab.com, leading to the glibc issue. This is clearly an easy mistake to make.

I ended up reverting the server to a recent snapshot and returning it to a known working status. I will be much more careful in future and leave myself some notes not to download the deb files from the packages.gitlab.com server without careful inspection.

I think apt wasn’t offering it as I hadn’t updated the package cache, so I thought I’d get it from the packages.gitlab.com server and install it as it was a security update, so I wanted to be covered ASAP.

1 Like

It should usually offer it, but I did have this situation once when I tried updating, it wouldn’t see any new packages. I was effectively stuck on 12.9.3. It was only when I went to the Gitlab install docs, and used the line to re-initialise the repository configuration, that it then saw updated packages. You can check/verify this, by doing the following:

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash

replace gitlab-ee with gitlab-ce if you are using that version, else you’ll end up adding the Enterprise repo instead of the Community one.

1 Like