Upgrade from 13.3.5 to 13.8.3 caused some corrupt repos
After Upgrading, everything looked very good as expected. But I found out that one repo was “missing”:
I went over this: how to fix gitlab ce migration empty repository error – Charlie Xiong but this did not match. I am still on default: /var/opt/gitlab/git-data/repositories
I found that the missing repo can be found at “https://gitlab/amxs/pm-module/activity”
but the rest said “empty”. Also /var/opt/gitlab/git-data/repositories/amxs/pm-module.git was GONE
I only found https://gitlab/admin/projects/amxs/pm-module page AFTER I tried to restore the bundle from backup:
git --mirror /var/opt/gitlab/backup/repositories/amxs/pm-module.bundle /var/opt/gitlab/git-data/repositories/amxs/pm-module.git
Then, the repo looked like it returned.
I found that:
gitlab-rake gitlab:storage:list_legacy_projects
lists all these Repos that could not be migrated and running especially one:
gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=4257 ID_TO=4257
it says:
Enqueueing storage migration of amxs/pm-module (ID=4257)…
Done!
But looking at sidekiq it fails due “RepositoryInUseError”:
Now I am lost. I did a local
git fsck on /var/opt/gitlab/git-data/repositories/amxs/pm-module
that was ok. and rescheduled the migration but it still fails with the same issue.
if you try to push into the repo you get:
❯ git push
remote:
remote: ========================================================================
remote:
remote: The repository is temporarily read-only. Please try again later.
remote:
remote: ========================================================================
remote:
fatal: Could not read from remote repository.
My assumption is: The migration went wrong at the upgrade and now the repo is possibly as well on the @hashed storage as well on the legacy one.
Do you have an Idea? Can I maybe find out the HASH of the Repo and look into the filesystem? thanks.