Gitlab repo GPG key no longer valid + Gitlab Inc. instructions on replacement don't seem to work on Rocky 8

vbarry · March 7, 2024, 10:09am

Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

In my Gitlab CE 16.9.1 installations running on Rocky 8.9

1). Gitlab repo GPG key no longer valid

My installation upgraded to 16.9.1 and many other previous version ok via my wrapper which, among other things, runs - yum check-update gitlab-ce | grep gitlab-ce | awk ‘{ print $2 }’

Today I get the following error from the above command when upgrading to 16.9.2
Error: Failed to download metadata for repo ‘gitlab_official_ce’: repomd.xml GPG signature verification error: Bad GPG signature

2). Gitlab Inc. instructions on replacement don’t work

I followed Cryptographic details related to `omnibus-gitlab` packages | GitLab as below

[myUser@myHost yum.repos.d]#  for pubring in /var/cache/dnf/gitlab_gitlab-?e-*/pubring
>  do
>    gpg --homedir $pubring --delete-key F6403F6544A38863DAA0B6E03F01618A51312F3F
>  done
gpg: WARNING: unsafe permissions on homedir '/var/cache/dnf/gitlab_gitlab-ce-2ebe8376d0fbb9f4/pubring'
gpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


pub  rsa4096/3F01618A51312F3F 2020-03-02 GitLab B.V. (package repository signing key) <packages@gitlab.com>

Delete this key from the keyring? (y/N) y
gpg: WARNING: unsafe permissions on homedir '/var/cache/dnf/gitlab_gitlab-ce-source-25fc24ba97d5cff1/pubring'
gpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


pub  rsa4096/3F01618A51312F3F 2020-03-02 GitLab B.V. (package repository signing key) <packages@gitlab.com>

Delete this key from the keyring? (y/N) y
gpg: WARNING: unsafe permissions on homedir '/var/cache/dnf/gitlab_gitlab-ce-source-c35465ca56c678d8/pubring'
gpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


pub  rsa4096/3F01618A51312F3F 2020-03-02 GitLab B.V. (package repository signing key) <packages@gitlab.com>

Delete this key from the keyring? (y/N) y
[myUser@myHost yum.repos.d]#

[myUser@myHost yum.repos.d]#  dnf check-update
Official repository for Gitlab                                                                                                                                                                                                                               656  B/s | 862  B     00:01
Official repository for Gitlab                                                                                                                                                                                                                                20 kB/s | 3.1 kB     00:00
Official repository for Gitlab                                                                                                                                                                                                                               873  B/s | 862  B     00:00
Error: Failed to download metadata for repo 'gitlab_official_ce': repomd.xml GPG signature verification error: Bad GPG signature

[myUser@myHost yum.repos.d]# curl "https://packages.gitlab.com/gpg.key" -o /tmp/omnibus_gitlab_gpg.key
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3191    0  3191    0     0  20856      0 --:--:-- --:--:-- --:--:-- 20856

[myUser@myHost yum.repos.d]# rpm --import /tmp/omnibus_gitlab_gpg.key

[myUser@myHost yum.repos.d]# rpm -q gpg-pubkey-f27eab47-60d4a67e --qf '%{name}-%{version}-%{release} --> %{summary}'
package gpg-pubkey-f27eab47-60d4a67e is not installed

[myUser@myHost yum.repos.d]# dnf check-update
Official repository for Gitlab                                                                                                                                                                                                                               662  B/s | 862  B     00:01
Official repository for Gitlab                                                                                                                                                                                                                                21 kB/s | 3.1 kB     00:00
Official repository for Gitlab                                                                                                                                                                                                                               704  B/s | 862  B     00:01
Error: Failed to download metadata for repo 'gitlab_official_ce': repomd.xml GPG signature verification error: Bad GPG signature
[myUser@myHost yum.repos.d]#

Steps to reproduce

When I changed repo_gpgcheck=1 to repo_gpgcheck=0, the upgrade worked

Configuration

Current config and I don’t want to run without checking the GPG on the repo:

[gitlab_official_ce]
name=Official repository for Gitlab
baseurl=https://packages.gitlab.com/gitlab/gitlab-ce/el/$releasever/$basearch
enabled=1
gpgcheck=0
repo_gpgcheck=1
gpgkey=https://packages.gitlab.com/gpg.key
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
sslverify=1

Versions

Please select whether options apply, and add the version information.

[ Y ] Self-managed
GitLab.com SaaS

Versions

1 host at 16.9.2 but other hosts at 16.9.1

Helpful resources

I checked here but there were no relevant matches

Thanks for taking the time to be thorough in your request, it really helps!

You’re welcome!

scmjerram · March 7, 2024, 10:54am

I am having similar problems. I get the following issue with update.

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic InRelease: The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <packages@gitlab.com>
W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu/dists/bionic/InRelease  The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <packages@gitlab.com>
W: Some index files failed to download. They have been ignored, or old ones used instead.

There is a 2020 article about updating the key

This leads to the same link posted above:

But… this directs me to running an awk script that needs root acces. Prefixing with sudo is not good enough!

awadmin@gitlab:~$ grep 'deb \[signed-by=' /etc/apt/sources.list.d/gitlab_gitlab-?e.list
deb [signed-by=/usr/share/keyrings/gitlab_gitlab-ee-archive-keyring.gpg] https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu/ bionic main
awadmin@gitlab:~$  awk '/deb \[signed-by=/{
>        pubkey = $2;
>        sub(/\[signed-by=/, "", pubkey);
>        sub(/\]$/, "", pubkey);
>        print pubkey
>      }' /etc/apt/sources.list.d/gitlab_gitlab-?e.list | \
>    while read line; do
>      curl -s "https://packages.gitlab.com/gpg.key" | gpg --dearmor > $line
>    done
-bash: /usr/share/keyrings/gitlab_gitlab-ee-archive-keyring.gpg: Permission denied
(23) Failed writing body

Adding sudo before awk does not help

dnsmichi · March 7, 2024, 11:17am

Maybe the comment in GitLab GPG expired today (#6701) · Issues · GitLab.org / omnibus-gitlab · GitLab helps.

vbarry · March 7, 2024, 1:12pm

I fixed my problem by running the following command

rm -rf /var/cache/dnf/gitlab

After running the above between steps 1 and 2 (for CentOS post 2020-04-06) at Cryptographic details related to `omnibus-gitlab` packages | GitLab, the following command no longer returned an error

dnf check-update

My automated updates work again - Thanks @dnsmichi - I found the DNF cache clean command via the link you posted

scmjerram · March 7, 2024, 1:14pm

Unfortunately we have already followed those instructions.

iwalker · March 7, 2024, 1:21pm

Someone had the issue on Debian, and they resolved it using my solution: Gitlab-ce Update Failing on bullseye - #2 by iwalker

Since Ubuntu is pretty much the same as Debian it should work for you as well.

scmjerram · March 7, 2024, 1:38pm

Thanks, this works, I think it completely reinstalled my link to the repository instead of just updating the key.

I’m still badgering our IT contractors to give me true root access to the server.