Unable to restore from self-compiled to omnibus (both in docker) -- "skipped restore" without context

Version: gitlab-ce-16.6.1 (docker) → source is self-compiled (sameersbn/docker-gitlab), target is docker-omnibus

I am trying to migrate gitlab from an unofficial container to the official one, following the official guides (e.g. this one). I have created a backup on the old container using the rake backup command, including repos and dbs (only skipping ci_secrets due to a different bug). Mind that some repos on the old container are in legacy format, i.e. not in @hashed, while others are hashed. That runs without a problem. After creating the new container and getting gitlab to run, copying the secrets and setting up the gitlab.rb file, I tried restoring the backup.

The restore command runs without errors, but creates hundreds of lines like the following:

{"command":"restore","gl_project_path":"<repo_path>.wiki","level":"warning","msg":"skipped restore","pid":1350,"relative_path":"@hashed/8b/d9/<hash>.wiki.git","storage_name":"default","time":"2024-01-11T20:18:38.918Z"}
{"command":"restore","gl_project_path":"<repo_path>","level":"warning","msg":"skipped restore","pid":1350,"relative_path":"@hashed/d8/74/<hash>.git","storage_name":"default","time":"2024-01-11T20:18:38.921Z"}
{"command":"restore","gl_project_path":"<repo_path>.wiki","level":"info","msg":"started restore","pid":1350,"relative_path":"@hashed/d8/74/<hash>.wiki.git","storage_name":"default","time":"2024-01-11T20:18:38.922Z"}
{"command":"restore","gl_project_path":"<repo_path>.wiki","level":"warning","msg":"skipped restore","pid":1350,"relative_path":"@hashed/d8/74/<hash>.wiki.git","storage_name":"default","time":"2024-01-11T20:18:38.925Z"}
{"command":"restore","gl_project_path":"snippets/1","level":"info","msg":"started restore","pid":1350,"relative_path":"@snippets/6b/86/<hash>.git","storage_name":"default","time":"2024-01-11T20:18:39.012Z"}
{"command":"restore","gl_project_path":"snippets/2","level":"info","msg":"started restore","pid":1350,"relative_path":"@snippets/d4/73/<hash>.git","storage_name":"default","time":"2024-01-11T20:18:39.014Z"}
{"command":"restore","gl_project_path":"snippets/1","level":"warning","msg":"skipped restore","pid":1350,"relative_path":"@snippets/6b/86/<hash>.git","storage_name":"default","time":"2024-01-11T20:18:39.017Z"}

There is absolutely no context about why these operations are skipped, and it results in a functional/complete project overview page, but every project you click on yields the message “This repo is empty”.

I also get the following errors, although I don’t believe they are related and are of lesser priority to me:

Snippet snippets/1 can't be restored: Error: Repository has more than one branch.
Snippet snippets/2 can't be restored: Error: Repository has more than one branch.
Snippet snippets/3 can't be restored: Error: Repository has more than one branch.
Snippet snippets/4 can't be restored: Error: Repository has more than one branch.
Snippet <repo_path>/snippets/5 can't be restored: Error: Repository has more than one branch.
Snippet <repo_path>/snippets/6 can't be restored: Error: Repository has more than one branch.
Snippet <repo_path>/snippets/7 can't be restored: Error: Repository has more than one branch.

gitlab-rake gitlab:check yields a successful result for all points (including migrations) except “All projects are in hashed storage”:

GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
  Try fixing it:
  Please migrate all projects to hashed storage
  as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
  For more information see:

This error is relatively useless however, as the migrations commands have not been available since version 14. The repositories folder is also completely empty except for the +gitaly folder and a hidden gitaly state file. No @hashed folder or legacy folder (both types are included in the backup, but were all skipped as seen above).

Manually moving the repository folders to the new repositories location fixes the “repo is empty” issue, but I’m not sure if that actually works for every aspect of these repos (like linked runners). Is there any information on why these restores might be skipped?

Have the same problem after having restored from backup. In my case, the hashed repo directory is now completely empty (no .git> and .wiki.git). During the restore, I saw similar warnings about “skipped restore”. Issues, Issue Board, Merge Requests, Milestones: everything is there. However, the list of commits for a merge request is empty, since there is no repo anymore.

I checked the backup tarfile, and there is a dump/001.bundle and dump/001.refs of reasonable size, which I guess contains the repo for this project.

My Gitlab version is 16.6.4.

You can try my temporary solution from here: Gitlab-ce-16.7.0 When I restored the backup, all the projects were shown as skipped restore. As a result, the entry page showed a project list, but all the projects contents were empty - #3 by xorwow

I cloned from the bundle that I had in my backup to repopulate the empty repo. Thanks @xorwow .

For anyone interested in the steps:

  • extract the bundle from the backup tar file (named backup_id_gitlab_backup.tar), usually named …/dump/001.bundle
  • navigate to the hashed repository dir (e.g. something like git-data/repositories/@hashed/??/??, where the ?? have to be replaced with the first two octets of the repo hash)
  • clone into a bare repo with git clone --bare path_to_bundle hash.git
  • fix ownership with chown -R 998:998 hash.git