Existence of `/var/opt/gitlab/gitlab-rails/shared/registry` after installing 14.8

wedwards · February 23, 2022, 1:01pm

I upgraded gitlab-ce from 14.7.0-ce.0 to 14.8.0-ce.0 yesterday. Ever since, my GitLab backup fails:

root@gitlab0:~# /usr/bin/gitlab-rake gitlab:backup:create --trace
** Invoke gitlab:backup:create (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:backup:create
rake aborted!
Errno::ENOENT: No such file or directory @ rb_check_realpath_internal - /var/opt/gitlab/gitlab-rails/shared/registry
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:16:in `realpath'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:16:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/registry.rb:10:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:29:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:29:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create

/var/opt/gitlab/gitlab-rails/shared/registry does indeed not exist. Looking at a backup from before the upgrade, it never has. GitLab 14.8 released with new SSH key types and security approval policies | GitLab does mention some registry-related changes, but nothing about it being a dependency now.

This commit changed L29 in manager.rb, which seems to be the culprit. This might be a regression, but before I create an issue I’m wondering … should /var/opt/gitlab/gitlab-rails/shared/registry exist? gitlab-ctl reconfigure does not create it.

philiaagape · February 23, 2022, 6:41pm

I have run into the exact same thing after upgrading from 14.7.0-ee to 14.8.0-ee on CentOS 7 (installed with omnibus packages). I’m wondering what the “best” solution is:

setting this in /etc/gitlab/gitlab.rb
gitlab_rails[‘registry_enabled’] = false
naively running
mkdir /var/opt/gitlab/gitlab-rails/shared/registry
modify the cron backup job to skip registry backups:
0 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1 SKIP=registry
something else…

iwalker · February 23, 2022, 6:45pm

It would be best to open an issue. Gitlab devs will not check here for problems. Only will take action for issues opened on the Gitlab repo.

14.8.1 has been released today though, not sure if that will resolve it. I wil update mine shortly and let you know if it does work.

philiaagape · February 23, 2022, 6:57pm

Running a manual backup fails even if I try to skip the registry backup:

/opt/gitlab/bin/gitlab-backup create SKIP=registry

And I’m unable to update to 14.8.1 since the backup during the update fails. I could skip that backup using

touch /etc/gitlab/skip-auto-backup

But that seems unsafe since I’d be upgrading without a backup (neither manual nor automatic backups work).

And the gitlab support page [ Support | GitLab ] seems to imply that a post on the gitlab forums (here) is the only way to get support for free self-hosted instances.

Let me know how you proceeded with upgrading to 14.8.1, if you got a manual backup to work before proceeding or just skipped backups all together. I’d also like to know if manual backups work after the upgrade to 14.8.1 and if so, if you had to change any settings regarding the registry (set registry_enabled = false, add SKIP=registry to the backup, or if you manually created the registry directory, or if that directory got created by the upgrade to 14.8.1).

Thanks

iwalker · February 23, 2022, 6:58pm

Yes, you cannot open a support ticket if a free user, this is why I will check mine now when I upgrade from 14.7.3 to 14.8.1. Will let you know.

What I mean is, that if it still persists, we need to open an issue on the Gitlab repository, so this project: GitLab.org / GitLab · GitLab if the case is that something has broke between 14.7.x and 14.8.x.

For info I made a manual backup first:

gitlab-backup create

so everything. And I will compare it with running the same command after upgrade.

iwalker · February 23, 2022, 7:13pm

So, these are my results. Manual backup before upgrade:

root@gitlab:~# gitlab-backup create
2022-02-23 19:50:03 +0100 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-02-23 19:50:10 +0100 -- done
2022-02-23 19:50:10 +0100 -- Dumping repositories ...
2022-02-23 19:50:11 +0100 -- done
2022-02-23 19:50:11 +0100 -- Dumping uploads ... 
2022-02-23 19:50:11 +0100 -- done
2022-02-23 19:50:11 +0100 -- Dumping builds ... 
2022-02-23 19:50:11 +0100 -- done
2022-02-23 19:50:11 +0100 -- Dumping artifacts ... 
2022-02-23 19:50:11 +0100 -- done
2022-02-23 19:50:11 +0100 -- Dumping pages ... 
2022-02-23 19:50:11 +0100 -- done
2022-02-23 19:50:11 +0100 -- Dumping lfs objects ... 
2022-02-23 19:54:58 +0100 -- done
2022-02-23 19:54:58 +0100 -- Dumping terraform states ... 
2022-02-23 19:54:58 +0100 -- done
2022-02-23 19:54:58 +0100 -- Dumping container registry images ... 
2022-02-23 19:54:58 +0100 -- [DISABLED]
2022-02-23 19:54:58 +0100 -- Dumping packages ... 
2022-02-23 19:54:58 +0100 -- done
Creating backup archive: 1645642498_2022_02_23_14.7.3_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.

as you can see by the filename, 14.7.3. Now the upgrade process, backup obviously also worked, since it was still 14.7.3 at this point, but for completeness:

root@gitlab:~# aptitude update && aptitude safe-upgrade
Hit http://security.debian.org/debian-security bullseye-security InRelease
Hit http://deb.debian.org/debian bullseye InRelease         
Hit http://deb.debian.org/debian bullseye-updates InRelease
Hit https://packages.gitlab.com/gitlab/gitlab-ce/debian bullseye InRelease
                                         
The following packages will be upgraded: 
  gitlab-ce libexpat1 
2 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 995 MB of archives. After unpacking 146 MB will be used.
Do you want to continue? [Y/n/?] 
Get: 1 http://security.debian.org/debian-security bullseye-security/main amd64 libexpat1 amd64 2.2.10-2+deb11u2 [98.2 kB]
Get: 2 https://packages.gitlab.com/gitlab/gitlab-ce/debian bullseye/main amd64 gitlab-ce amd64 14.8.1-ce.0 [995 MB]
Fetched 995 MB in 44s (22.8 MB/s)                                                                                                            
Reading changelogs... Done
(Reading database ... 167263 files and directories currently installed.)
Preparing to unpack .../libexpat1_2.2.10-2+deb11u2_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.10-2+deb11u2) over (2.2.10-2+deb11u1) ...
Preparing to unpack .../gitlab-ce_14.8.1-ce.0_amd64.deb ...
gitlab preinstall: Checking for unmigrated data on legacy storage
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
2022-02-23 19:58:47 +0100 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-02-23 19:58:49 +0100 -- done
2022-02-23 19:58:49 +0100 -- Dumping repositories ...
2022-02-23 19:58:49 +0100 -- [SKIPPED]
2022-02-23 19:58:49 +0100 -- Dumping uploads ... 
2022-02-23 19:58:49 +0100 -- [SKIPPED]
2022-02-23 19:58:49 +0100 -- Dumping builds ... 
2022-02-23 19:58:49 +0100 -- [SKIPPED]
2022-02-23 19:58:49 +0100 -- Dumping artifacts ... 
2022-02-23 19:58:49 +0100 -- [SKIPPED]
2022-02-23 19:58:49 +0100 -- Dumping pages ... 
2022-02-23 19:58:49 +0100 -- [SKIPPED]
2022-02-23 19:58:49 +0100 -- Dumping lfs objects ... 
2022-02-23 19:58:49 +0100 -- [SKIPPED]
2022-02-23 19:58:49 +0100 -- Dumping terraform states ... 
2022-02-23 19:58:49 +0100 -- [SKIPPED]
2022-02-23 19:58:49 +0100 -- Dumping container registry images ... 
2022-02-23 19:58:49 +0100 -- [DISABLED]
2022-02-23 19:58:49 +0100 -- Dumping packages ... 
2022-02-23 19:58:49 +0100 -- [SKIPPED]
Creating backup archive: 1645642729_2022_02_23_14.7.3_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.

Now, the backup after the upgrade was done:

2022-02-23 20:03:47 +0100 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-02-23 20:03:49 +0100 -- done
2022-02-23 20:03:49 +0100 -- Dumping repositories ... 
2022-02-23 20:03:49 +0100 -- done
2022-02-23 20:03:49 +0100 -- Dumping uploads ... 
2022-02-23 20:03:49 +0100 -- done
2022-02-23 20:03:49 +0100 -- Dumping builds ... 
2022-02-23 20:03:49 +0100 -- done
2022-02-23 20:03:49 +0100 -- Dumping artifacts ... 
2022-02-23 20:03:49 +0100 -- done
2022-02-23 20:03:49 +0100 -- Dumping pages ... 
2022-02-23 20:03:49 +0100 -- done
2022-02-23 20:03:49 +0100 -- Dumping lfs objects ... 
2022-02-23 20:08:49 +0100 -- done
2022-02-23 20:08:49 +0100 -- Dumping terraform states ... 
2022-02-23 20:08:49 +0100 -- done
2022-02-23 20:08:49 +0100 -- Dumping container registry images ... 
2022-02-23 20:08:49 +0100 -- [DISABLED]
2022-02-23 20:08:49 +0100 -- Dumping packages ... 
2022-02-23 20:08:49 +0100 -- done
Creating backup archive: 1645643329_2022_02_23_14.8.1_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.

so this has worked for me (you can see 14.8.1 in the backup filename), obviously something was wrong in 14.8.0 that was resolved in 14.8.1. If you cannot do a backup for 14.8.0 I would suggest doing the skip option that you suggested, but if possible make a snapshot of your server if it is a virtual machine, vps, etc. That way you can always revert to the snapshot if it goes wrong.

Alternatively, if you cannot do that, then the best way would be to completely stop Gitlab, and then make a backup of the following locations:

/etc/gitlab
/opt/gitlab
/var/opt/gitlab
/lib/systemd/system/gitlab-runsvdir.service

At least if something goes wrong, you can then restore these locations.

philiaagape · February 23, 2022, 7:46pm

Thank you kindly for the details! I’m most encouraged that it is skipping the container registry during backups.

I am running a virtual machine, so I’ll just use a snapshot as my restore point backup, then upgrade after touching /etc/gitlab/skip-auto-backup

[UPDATE]:
This all went well and I am now able to run “gitlab-backup create” without issue. It does mention that it is skipping the container registry dump.

The one thing to point out with the update to 14.8.1 from 14.8.0 is this output during the update process:

Recipe: registry::disable
  * service[registry] action nothing (skipped due to action :nothing)
  * runit_service[registry] action disable
    * ruby_block[disable registry] action run (skipped due to only_if)
     (up to date)

This was the only thing that mentioned the world “registry” in the output of the update. It looks like they disable the registry by default now (maybe? it looks like it also says it skipped all the actions).

I can confirm that the directory “/var/opt/gitlab/gitlab-rails/shared/registry” still does not exist, and /etc/gitlab/gitlab.rb does NOT set gitlab_rails[‘registry_enabled’] to anything (either true or false).

For anyone else following this, the last step I did was to remove the file /etc/gitlab/skip-auto-backup so that it does take backups during updates in the future.

wedwards · February 23, 2022, 8:22pm

14.8.1 contains Stop backup files from requiring directories to exist when skipped (!81098) · Merge requests · GitLab.org / GitLab · GitLab, which says:

Calls to File.realpath where the directory does not exist will raise an exception. Since all backup tasks are initialized no matter if they are skipped or not, we need to stop calling realpath within the initializer. Instead here we are lazily initializing realpath as required.

wedwards · February 23, 2022, 8:27pm

This is actually a duplicate of Backup rake task is failing on nodes with services disabled (#353242) · Issues · GitLab.org / GitLab · GitLab. It didn’t come up in my earlier searches.

it_apkl · February 25, 2022, 10:11am

Running Debian Buster 10 with gitlab-ce installed with apt I had the same problem upgrading from 14.8.0 to 14.8.1.

Quick fix

Have a peek at the parent-folder where missing folder should be. Note what rights and owner the other folders have:

$ sudo ls -alh /var/opt/gitlab/gitlab-rails/shared
insgesamt 56K
drwxr-x--x 14 git gitlab-www 4.0K Feb 25 09:59 .
drwxr-xr-x  9 git root       4.0K Feb 25 10:02 ..
drwx------  2 git git        4.0K Mai 16  2020 artifacts
drwx------  2 git git        4.0K Mai 16  2020 cache
drwx------  2 git git        4.0K Feb 23 08:33 ci_secure_files
drwx------  2 git git        4.0K Mai 16  2020 dependency_proxy
drwx------  2 git git        4.0K Jan 25  2021 encrypted_settings
drwx------  2 git git        4.0K Mai 16  2020 external-diffs
drwx------  2 git git        4.0K Mai 16  2020 lfs-objects
drwx------  2 git git        4.0K Mai 16  2020 packages
drwxr-x---  2 git gitlab-www 4.0K Mai 16  2020 pages
drwx------  2 git git        4.0K Mai 16  2020 terraform_state
drwx------  2 git git        4.0K Mai 16  2020 tmp

So I took the following steps to create the folder and adjust the rights:

sudo mkdir /var/opt/gitlab/gitlab-rails/shared/registry
sudo chown git:git /var/opt/gitlab/gitlab-rails/shared/registry
sudo chmod 700 /var/opt/gitlab/gitlab-rails/shared/registry

After that the upgrade worked like a charm.

mbvqt · February 25, 2022, 11:41am

/var/opt/gitlab/gitlab-rails/shared/registry with 700 for git:git worked for me on a very similar install. Thank you!

Topic		Replies	Views
/var/opt/gitlab/registry/storage permissions? How to Use GitLab backup	1	1488	February 23, 2022
Error with gitlab backup after GItlab CE 14.3.6 version upgrade Upgrade	3	924	November 19, 2022
Backup rake task failing following system upgrade Upgrade	3	992	March 12, 2019
Can't backup the registry Self-managed backup	1	930	October 16, 2020
Error during restoring registry backup How to Use GitLab	1	962	October 20, 2017

Existence of `/var/opt/gitlab/gitlab-rails/shared/registry` after installing 14.8

Quick fix

Related topics