Failed backup (rake aborted), failed update (key error)

How to Use GitLab Self-managed
1

Configuration

I’m running gitlab CE v16.8.2 under Ubuntu 22.04.

Problem to solve

I just had a backup fail with an error that I don’t recognize (log below).

I’m not sure if it’s related, but updating also failed with a key error (log below).

Any suggestions?

Failed backup

$ gitlab-rake gitlab:backup:create STRATEGY=copy
…
2024-03-03 18:19:40 UTC – Deleting tar staging files …
2024-03-03 18:19:40 UTC – Cleaning up /var/opt/gitlab/backups/db
2024-03-03 18:19:40 UTC – Cleaning up /var/opt/gitlab/backups/repositories
2024-03-03 18:19:44 UTC – Deleting tar staging files … done
2024-03-03 18:19:44 UTC – Deleting backups/tmp …
2024-03-03 18:19:44 UTC – Deleting backups/tmp … done
2024-03-03 18:19:44 UTC – Deleting backup and restore PID file … done
rake aborted!
Backup::Error: gitaly-backup exit status 1
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:49:in finish!' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:30:in dump’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:71:in run_create_task' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:227:in block in run_all_create_tasks’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:226:in each_key' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:226:in run_all_create_tasks’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:47:in create' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:13:in block in create_backup’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:62:in lock_backup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in create_backup’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:101:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:25:in load’
/opt/gitlab/embedded/bin/bundle:25:in `’
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)

Failed update

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu jammy InRelease: The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) packages@gitlab.com
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.gitlab.com/runner/gitlab-runner/ubuntu jammy InRelease: The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) packages@gitlab.com
W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/dists/jammy/InRelease The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) packages@gitlab.com
W: Failed to fetch https://packages.gitlab.com/runner/gitlab-runner/ubuntu/dists/jammy/InRelease The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) packages@gitlab.com
W: Some index files failed to download. They have been ignored, or old ones used instead.

2

The GPG Key for the update package has been extended and this error should be resolved -

3

The documentation calls out an issue with backups on 16.8.x and PgBouncer. If your environment has PgBouncer the fix is to bypass the same or upgrade to 16.9.

4

I had a similar problem with my GitLab CE where backups failed and updates wouldn’t work because of a key error. The backup issue was due to a problem with Gitaly, the part of GitLab that manages Git repositories. Checking and fixing Gitaly sorted it out. For the update issue, it was because the GPG key for GitLab’s package repository had expired. I fixed it by getting the new GPG key from GitLab’s site and updating it on my system. Both times, GitLab’s official guides really helped me find solutions.

5

Thank you for the help; running:

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

replaced the keys, then after update/upgrade to 16.9.2 the backup completed.

Neil