Cannot update gitlab after OS upgrade to Ubuntu 22.04 "Jammy"

I’ve done an OS upgrade today from Ubuntu 20.04 to 22.04 on a bare-metal server. Everything went fine, except updating GitLab APT repos.

I’ve re-downloaded the repository script and re-ran it, but when I run apt update I am confronted with these warnings and errors:

W: GPG error: https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F
E: The repository 'https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu jammy InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

A quick google of NO_PUBKEY produces about 30 different articles, all saying to run either of the following commands:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3F01618A51312F3F
curl --silent https://packages.gitlab.com/gpg.key | sudo apt-key add -

I’ve tried both, and no effect. Running apt update still generates these GPG errors. On top of that, apt-key is deprecated, and they’re supposed to be done manually(?) in trusted.gpg.d instead. A reboot didn’t do anything either.

I think that somewhere I need to put the correct pubkey and overwrite some old outdated broken one, and I don’t know where it is or how to do that.

None of the articles I found elaborated on this whatsoever, or what to do when their GPG command doesn’t work.

Earlier today I did a test upgrade on a VM, which didn’t encounter any of these issues as the VM wasn’t over a year old.

Currently the GitLab install is valid and functional, I just can’t get apt to trust any third-party repos anymore. Help would be greatly appreciated.

Maintaining /etc/apt/trusted.gpg.d has AFAIK never been official advice, it’s more or less what apt-key did, and was just a way to avoid doing changes to files in sources.list.d.

What does your /etc/apt/sources.list.d/gitlab (or something like that, I don’t remember what that file is called, on the GitLab servers I maintain we have chef put it in /etc/apt/sources.list) look like? If it says (and it should) ...signed-by=<file>, try replacing (save a copy in case something goes wrong) <file> with the contents from https://packages.gitlab.com/gpg.key

The sources file /etc/apt/sources.list.d/gitlab_gitlab-ee.list contains these lines (excluding comments):

deb [signed-by=/usr/share/keyrings/gitlab_gitlab-ee-archive-keyring.gpg] https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu/ jammy main
deb-src [signed-by=/usr/share/keyrings/gitlab_gitlab-ee-archive-keyring.gpg] https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu/ jammy main

The contents of /usr/share/keyrings/gitlab_gitlab-ee-archive-keyring.gpg appears to be a binary file, either that or the file is corrupted. Looking at other files in the directory, all of them appear to be binary (though when running apt update only the GitLab repo fails).

I moved the current one to back it up, curled the linked file above into the same contents, and ran chmod 644 to match the others. No luck. Running apt update still fails on GitLab.

Figured it out. I had UMASK set to 027, which worked just fine on Focal. But now on Jammy it seems that gets in the way.

Set it back to 022 (in /etc/login.defs), rebooted, re-downloaded and re-ran the install script, and I can now run apt update.

Those files are supposed to be binary, I should probably have said so when I asked what it looked like.

Setting umask just means some files get a different set of permissions, I would be interested in knowing which it was (and what the permissions that did not/did work were), but with so many changes I guess you can’t tell.