Cannot delete project w/o repo

Hello. On my self-hosted GitLab CE 13.12.15 I have group with Git LFS status: Enabled for all projects. Inside this group only two projects. I need to delete one of this projects because i’m stuck on update to 14.0 - update process says this project in legacy storage yet. I cannot migrate it to hashed storage (Problem with Hashed / Legacy storage migration)

Our developers says this repository (project) not needed anymore, so i try to delete it. But in admin area → projects → %project_name% i see repository: does not exist, so direct link to project return 404 - so i cannot access project settings.

I already tried:

  1. delete from group page, but got 404 - my browser tried to open Project page
  2. delete through API request, but got {"message":"404 Project Not Found"}. Anyway, GET request by project id return the same
  3. I’d tried to create empty repo and place %projectName%/.git inside my lfs-repositories directory as legacy storage.
  4. similarly to (3) i’d tried to place this empty repo inside non-LFS repositories directory

One strange thing - in admin area → groups → %groupName% this project marked as consuming 72.1 GB of space. All of this space is outdated artifacts, job logs and so on. I already deleted this stuff with Jobs artifacts administration | GitLab and saw freed space on my disks. But group page says this project still consume 72.1 GB

Maybe someone have gitlab-rails console snippet for project deletion?

Ok, I found GitLab Rails Console Cheat Sheet | GitLab and seems it may help me. In sidekiq current log file i found next string -

{"severity":"INFO","time":"2021-11-15T11:41:27.413Z","class":"ProjectDestroyWorker","args":["75","336","[FILTERED]"],"retry":3,"queue":"project_destroy","version":0,"backtrace":5,"jid":"c61ce41dc66204f52cdca434","created_at":"2021-11-15T11:41:27.389Z","correlation_id":"c91942234b992b1f65a0936cf3219bb5","enqueued_at":"2021-11-15T11:41:27.411Z","pid":1102,"message":"ProjectDestroyWorker JID-c61ce41dc66204f52cdca434: start","job_status":"start","scheduling_latency_s":0.00197,"job_size_bytes":262}
{"severity":"INFO","time":"2021-11-15T12:19:08.696Z","class":"DeleteStoredFilesWorker","args":["Uploads::Local","[FILTERED]"],"retry":3,"queue":"delete_stored_files","version":0,"jid":"97cbb79bd490a252f7b69a0a","created_at":"2021-11-15T12:19:08.685Z","meta.caller_id":"ProjectDestroyWorker","meta.feature_category":"source_code_management","correlation_id":"c91942234b992b1f65a0936cf3219bb5","enqueued_at":"2021-11-15T12:19:08.692Z","pid":1102,"message":"DeleteStoredFilesWorker JID-97cbb79bd490a252f7b69a0a: start","job_status":"start","scheduling_latency_s":0.003841,"job_size_bytes":6799}
{"severity":"INFO","time":"2021-11-15T12:19:08.705Z","class":"ProjectDestroyWorker","args":["75","336","[FILTERED]"],"retry":3,"queue":"project_destroy","version":0,"backtrace":5,"jid":"c61ce41dc66204f52cdca434","created_at":"2021-11-15T11:41:27.389Z","correlation_id":"c91942234b992b1f65a0936cf3219bb5","enqueued_at":"2021-11-15T11:41:27.411Z","pid":1102,"message":"ProjectDestroyWorker JID-c61ce41dc66204f52cdca434: done: 2261.291972 sec","job_status":"done","scheduling_latency_s":0.00197,"job_size_bytes":262,"cpu_s":1247.702735,"gitaly_calls":38,"gitaly_duration_s":0.334682,"mem_objects":164989760,"mem_bytes":6391489392,"mem_mallocs":34145351,"redis_calls":349,"redis_duration_s":0.240329,"redis_read_bytes":1031,"redis_write_bytes":59715,"redis_cache_calls":340,"redis_cache_duration_s":0.236588,"redis_cache_read_bytes":905,"redis_cache_write_bytes":49065,"redis_queues_calls":9,"redis_queues_duration_s":0.003741,"redis_queues_read_bytes":126,"redis_queues_write_bytes":10650,"db_count":638201,"db_write_count":315749,"db_cached_count":10,"duration_s":2261.291972,"completed_at":"2021-11-15T12:19:08.705Z","db_duration_s":1196.657504}
{"severity":"INFO","time":"2021-11-15T12:19:08.867Z","class":"DeleteStoredFilesWorker","args":["Uploads::Local","[FILTERED]"],"retry":3,"queue":"delete_stored_files","version":0,"jid":"97cbb79bd490a252f7b69a0a","created_at":"2021-11-15T12:19:08.685Z","meta.caller_id":"ProjectDestroyWorker","meta.feature_category":"source_code_management","correlation_id":"c91942234b992b1f65a0936cf3219bb5","enqueued_at":"2021-11-15T12:19:08.692Z","pid":1102,"message":"DeleteStoredFilesWorker JID-97cbb79bd490a252f7b69a0a: done: 0.171027 sec","job_status":"done","scheduling_latency_s":0.003841,"job_size_bytes":6799,"cpu_s":0.026398,"mem_objects":818,"mem_bytes":60544,"mem_mallocs":183,"redis_calls":2,"redis_duration_s":0.037637,"redis_read_bytes":2,"redis_write_bytes":188,"redis_queues_calls":2,"redis_queues_duration_s":0.037637,"redis_queues_read_bytes":2,"redis_queues_write_bytes":188,"db_count":0,"db_write_count":0,"db_cached_count":0,"duration_s":0.171027,"completed_at":"2021-11-15T12:19:08.867Z","db_duration_s":0}

The project is still showing on the group page. The API DELETE request returns 202 {" message ":" 202 Accepted "} now and I can remove the project from the group settings. Api and web requests show similar behavior - 4 entries in the sidekiq log that the deletion completed successfully. But the project is still showing and

# gitlab-rake gitlab:storage:list_legacy_projects
* Found 1 projects using Legacy Storage
  - awesomerepo / awesomerepo (id: 75)