A number of hashed_storage_project_migrate jobs failed / Projects show "The repository for this project does not exist"

This is somewhat related to this earlier discussion and this issue.

We have a small number of repositories that failed to migrate to hashed storage after upgrading GitLab. The error we’re seeing is a Projects::HashedStorage::RepositoryInUseError with the message “Target repository ‘group/project’ cannot be made read-only: Repository already read-only”

Re-running the Sidekiq jobs results in the same error.

Does anyone have some advice on how to proceed?

I have the same problem.
And it seems because of this I cannot get back the disappeared repositories.

I could solve with this

It was a huge hack.

Thanks, I had also found that Issue and was keeping it aside as a last resort option. After all, messing with stuff directly through the Rails Console is a bit dangerous. Your message gave me the confidence to try it myself :slight_smile:

For anyone else that encounters this, here is how I solved it. Bear in mind that there may be other causes for this or similar errors (in my case it seemed to really only be that the repository_read_only flag was already set, for whatever reason). Proceed at your own risk.

If you don’t already know which repositories/projects are affected, you should see a failed Sidekiq job for each under Admin Area > Monitoring > Background Jobs > Dead. The Repository storage Rake tasks can also come in handy to confirm the no. and list the projects stuck in legacy storage.

  1. Start a Rails console session
  2. Find the project by name: p = Project.find_by_name("<project-name>")
  3. Confirm that the repository is in fact read-only: p.repository_read_only
  4. Unset the repository_read_only flag: p.update!(repository_read_only:nil)
  5. Retry the corresponding Sidekiq job in the Admin Area (for me it would succeed almost instantly - the dead job count decreased and I never saw a running job)
  6. Rinse and repeat from step 2.

I could solve it. Thank you for the detailed info.