Problem with Hashed / Legacy storage migration

Hello. I’ve try to update gitlab-ce from 13.12.15 to 14.0.12 and get legacy storage warning. Following docs i tried to execute
gitlab-rake gitlab:storage:migrate_to_hashed
Command output -
Enqueuing migration of 1 projects in batches of 200. Done!
But gitlab-rake gitlab:storage:list_legacy_projects still show 1 project in legacy storage. Also, several (45) attachments in legacy storage too.

In sidekiq current log i isn’t see any errors -

{"severity":"INFO","time":"2021-11-12T19:13:46.222Z","class":"HashedStorage::ProjectMigrateWorker","args":["75"],"retry":3,"queue":"hashed_storage:hashed_storage_project_migrate","version":0,"queue_namespace":"hashed_storage","jid":"14fbdecc76cd8c63cee2c6ba","created_at":"2021-11-12T19:13:46.218Z","meta.caller_id":"HashedStorage::MigratorWorker","meta.feature_category":"source_code_management","correlation_id":"85aa1403ac8b4ec5152a5cf66688b929","enqueued_at":"2021-11-12T19:13:46.220Z","pid":1102,"message":"HashedStorage::ProjectMigrateWorker JID-14fbdecc76cd8c63cee2c6ba: start","job_status":"start","scheduling_latency_s":0.002798,"job_size_bytes":419}
{"severity":"INFO","time":"2021-11-12T19:13:46.225Z","class":"HashedStorage::MigratorWorker","args":["75","75"],"retry":3,"queue":"hashed_storage:hashed_storage_migrator","version":0,"queue_namespace":"hashed_storage","jid":"55b26df2d5c13c1870cd9536","created_at":"2021-11-12T19:13:46.151Z","correlation_id":"85aa1403ac8b4ec5152a5cf66688b929","enqueued_at":"2021-11-12T19:13:46.188Z","pid":1102,"message":"HashedStorage::MigratorWorker JID-55b26df2d5c13c1870cd9536: done: 0.034644 sec","job_status":"done","scheduling_latency_s":0.002273,"job_size_bytes":312,"cpu_s":0.018412,"mem_objects":3019,"mem_bytes":128832,"mem_mallocs":580,"redis_calls":7,"redis_duration_s":0.004232,"redis_read_bytes":263,"redis_write_bytes":1383,"redis_cache_calls":1,"redis_cache_duration_s":0.000494,"redis_cache_read_bytes":203,"redis_cache_write_bytes":55,"redis_queues_calls":3,"redis_queues_duration_s":0.002634,"redis_queues_read_bytes":60,"redis_queues_write_bytes":1188,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.001104,"redis_shared_state_write_bytes":140,"db_count":2,"db_write_count":0,"db_cached_count":0,"duration_s":0.034644,"completed_at":"2021-11-12T19:13:46.225Z","db_duration_s":0.010333}
{"severity":"INFO","time":"2021-11-12T19:13:46.246Z","class":"HashedStorage::ProjectMigrateWorker","args":["75"],"retry":3,"queue":"hashed_storage:hashed_storage_project_migrate","version":0,"queue_namespace":"hashed_storage","jid":"14fbdecc76cd8c63cee2c6ba","created_at":"2021-11-12T19:13:46.218Z","meta.caller_id":"HashedStorage::MigratorWorker","meta.feature_category":"source_code_management","correlation_id":"85aa1403ac8b4ec5152a5cf66688b929","enqueued_at":"2021-11-12T19:13:46.220Z","pid":1102,"message":"HashedStorage::ProjectMigrateWorker JID-14fbdecc76cd8c63cee2c6ba: done: 0.023355 sec","job_status":"done","scheduling_latency_s":0.002798,"job_size_bytes":419,"cpu_s":0.008957,"mem_objects":1140,"mem_bytes":59808,"mem_mallocs":243,"redis_calls":4,"redis_duration_s":0.003694,"redis_read_bytes":4,"redis_write_bytes":529,"redis_queues_calls":2,"redis_queues_duration_s":0.000882,"redis_queues_read_bytes":2,"redis_queues_write_bytes":214,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.002812,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":315,"db_count":1,"db_write_count":0,"db_cached_count":0,"duration_s":0.023355,"completed_at":"2021-11-12T19:13:46.246Z","db_duration_s":0.003143}

In admin area → Projects i see repository: **does not exist**, in repositories directory on disk correspond folder empty.

I’ve tried to use workaround fix - Fix inconsistent storage migration between legacy/hashed storage ($2039252) · Snippets · Snippets · GitLab but i’m still there

Looks like my repo already migrated to hashed storage (because of empty dir), but gitlab don’t know about it. How can i solve this?

This is single-node omnibus gitlab installation on ubuntu 18.04

Additional info - repository is not read only:


irb(main):001:0> p = Project.find_by_name("awesomerepo")

=> #<Project id:75 awesomerepo/awesomerepo>>

irb(main):002:0> p.repository_read_only

=> nil

Secondly, gitlab-rake gitlab:git:fsck shows an error:


- awesomerepo/awesomerepo on default: 5:GetRepoPath: not a git repository: "/opt/git-data/repositories/awesomerepo/awesomerepo.git". debug_error_string:{"created":"@1636775667.938608328","description":"Error received from peer unix:/var/opt/gitlab/gitaly/gitaly.socket","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"GetRepoPath: not a git repository: "/opt/git-data/repositories/awesomerepo/awesomerepo.git"","grpc_status":5}

This error shows that directory under repositories empty, i still think all data already in hashed storage…

Hello,

Do you know if the workaround script actually did something - i.e. did it output “Fixing ID:#…” at all?

A few people, including myself, had a similar problem here, but it looks like yours won’t be as easy to solve. The migration Sidekiq job seems to be completing…

Some questions that might help to narrow it down:
What is the value of the project’s storage_version property now?
Can you actually find the project in hashed storage?

Clearly there is no Git repo at the old path anymore. If it’s in hashed storage and everything checks out, then I think you’ll have to try and figure out where the old repo path is stored and fix it manually (or manually perform some of the steps from the workaround script to correct the config).

Thanks for your reply.
No, the workaround script did not output anything. And you’re right, the problematic repo is missing from any of the storage paths - I can’t find it in the legacy or hashed storage. It seems this repo disappeared someday - and I don’t know when.
As I posted in other topic Cannot delete project w/o repo our developers sayed this repository (project) not needed anymore, so i tried to delete it.
After all, gitlab-rails destroy project snippet restored this project w/o repository - so i can accessing issues, pipelines and so on - except repository, obviously. Project successfully moved to hashed storage and upgrade proccess continued