A number of hashed_storage_project_migrate jobs failed / Projects show "The repository for this project does not exist"

This is somewhat related to this earlier discussion and this issue.

We have a small number of repositories that failed to migrate to hashed storage after upgrading GitLab. The error we’re seeing is a Projects::HashedStorage::RepositoryInUseError with the message “Target repository ‘group/project’ cannot be made read-only: Repository already read-only”

Re-running the Sidekiq jobs results in the same error.

Does anyone have some advice on how to proceed?

1 Like

I have the same problem.
And it seems because of this I cannot get back the disappeared repositories.

I could solve with this

It was a huge hack.

1 Like

Thanks, I had also found that Issue and was keeping it aside as a last resort option. After all, messing with stuff directly through the Rails Console is a bit dangerous. Your message gave me the confidence to try it myself :slight_smile:

For anyone else that encounters this, here is how I solved it. Bear in mind that there may be other causes for this or similar errors (in my case it seemed to really only be that the repository_read_only flag was already set, for whatever reason). Proceed at your own risk.

If you don’t already know which repositories/projects are affected, you should see a failed Sidekiq job for each under Admin Area > Monitoring > Background Jobs > Dead. The Repository storage Rake tasks can also come in handy to confirm the no. and list the projects stuck in legacy storage.

  1. Start a Rails console session
  2. Find the project by name: p = Project.find_by_name("<project-name>")
  3. Confirm that the repository is in fact read-only: p.repository_read_only
  4. Unset the repository_read_only flag: p.update!(repository_read_only:nil)
  5. Retry the corresponding Sidekiq job in the Admin Area (for me it would succeed almost instantly - the dead job count decreased and I never saw a running job)
  6. Rinse and repeat from step 2.
4 Likes

I could solve it. Thank you for the detailed info.

2 Likes

Took me a day to reach out here. Dunno why searching this soecific error does not quicker lead you tocthis thread.
But I am so glad you fund that hack.
It happened to some repos and now I am happy again! :slight_smile:

3 Likes

@maltem-za this fix really worked for me. Thank you!

If you have hundreds of repos, this will save you some time!

Create a file /tmp/fix_readonly.rb with the following contents:

# Find all projects that GitLab thinks is in legacy storage
Project.without_storage_feature(:repository).find_each(batch_size: 10) do |p|
        # Clear read only
        p.update!(repository_read_only:nil)
end

Then run it:

gitlab-rails runner /tmp/fix_readonly.rb

1 Like