A number of hashed_storage_project_migrate jobs failed / Projects show "The repository for this project does not exist"

How to Use GitLab Upgrade
1

This is somewhat related to this earlier discussion and this issue.

We have a small number of repositories that failed to migrate to hashed storage after upgrading GitLab. The error we’re seeing is a Projects::HashedStorage::RepositoryInUseError with the message “Target repository ‘group/project’ cannot be made read-only: Repository already read-only”

Re-running the Sidekiq jobs results in the same error.

Does anyone have some advice on how to proceed?

1 Like
2

I have the same problem.
And it seems because of this I cannot get back the disappeared repositories.

3

I could solve with this

It was a huge hack.

2 Likes
4

Thanks, I had also found that Issue and was keeping it aside as a last resort option. After all, messing with stuff directly through the Rails Console is a bit dangerous. Your message gave me the confidence to try it myself :slight_smile:

For anyone else that encounters this, here is how I solved it. Bear in mind that there may be other causes for this or similar errors (in my case it seemed to really only be that the repository_read_only flag was already set, for whatever reason). Proceed at your own risk.

If you don’t already know which repositories/projects are affected, you should see a failed Sidekiq job for each under Admin Area > Monitoring > Background Jobs > Dead. The Repository storage Rake tasks can also come in handy to confirm the no. and list the projects stuck in legacy storage.

  1. Start a Rails console session
  2. Find the project by name: p = Project.find_by_name("<project-name>")
  3. Confirm that the repository is in fact read-only: p.repository_read_only
  4. Unset the repository_read_only flag: p.update!(repository_read_only:nil)
  5. Retry the corresponding Sidekiq job in the Admin Area (for me it would succeed almost instantly - the dead job count decreased and I never saw a running job)
  6. Rinse and repeat from step 2.
6 Likes
5

I could solve it. Thank you for the detailed info.

3 Likes
6

Took me a day to reach out here. Dunno why searching this soecific error does not quicker lead you tocthis thread.
But I am so glad you fund that hack.
It happened to some repos and now I am happy again! :slight_smile:

3 Likes
7

@maltem-za this fix really worked for me. Thank you!

If you have hundreds of repos, this will save you some time!

Create a file /tmp/fix_readonly.rb with the following contents:

# Find all projects that GitLab thinks is in legacy storage
Project.without_storage_feature(:repository).find_each(batch_size: 10) do |p|
        # Clear read only
        p.update!(repository_read_only:nil)
end

Then run it:

gitlab-rails runner /tmp/fix_readonly.rb

2 Likes
8

Thank you very much! I solved my problem using this tip!

1 Like
9

We still have the same case without a solution over here: Error on Gitlab CE version 13.6 No repository - #2 by vhristev

Nothing worked for me:

  • Clear cache
  • Migrate legacy storage to hashes
  • Clear registration tokens.

Still cannot see repos in UI but the data is on the server.