Gitlab suddenly stopped serving LFS files from one of my repos

After about a year of inactivity on one of my reposiitories (and a few gitlab version updates) gitlab suddenly stopped serving the LFS content.
It gives the common - [404] Object does not exist on the server or you don't have permissions to access it.

LFS was working perfectly fine for this repository.
There was months of development, without issue.
All the CI/CD jobs were running fine.

Now, i return to this project about a year later… I cannot clone it!
ALL the lfs obhects are gone (404).
The very strange thing is… That I can download them just fine from the Web UI !?
If I download the project as a ZIP, the zip contains the LFS content…
But when i git clone via command line… 404!?
If i Re-Run a CI/CD Job, if fails, where before it had succeeded!

I can log into my locally local LFS storage, and i can see all the files that it says are gone !?

Other projects created before and after the faulty one continue to function perfectly.
It is just this one project that is giving problems.

How can I restore, or export, this project?
Where should I start debugging?

This is a private in house server, (Omnibus install) not accessable outside of our LAN.

This is just one corripted? project so far… But I am really starting to worry about the many others.
Its really scaring me than an un-touched project can become silently broken. losing a lot of work.

EDIT:

Update… after upgrading to the latest version of gitlab… and re-booting… the web UI is nolonger able to access my LFS content either… I suppose it was cached or something!??

So, now im in a worse position… gitlab has lost one my out projects…

We use LFS for game assets… there are hundreds of files / file revisions… its going to take me forever to re-create a git repo from a file-server dump, and broken gitlab repo

any ideas… anyone?
Ill just be under my desk, having a little cry

I have no idea how it got in this state… but a co-worker, far more gifted than myself discovered that an internal lfs database was empty…

the solution was to get a list of the lfs objects by processing the output of git lfs fetch --all 2> error.txt.
and re-adding them to the database…

LfsObjectsProject.create(project_id: 337, lfs_object_id: LfsObject.find_by(oid: '731225b9aa4cd08e0142b3509f25616e0a82500c045ad06434347e8af273f93d').id)
LfsObjectsProject.create(project_id: 337, lfs_object_id: LfsObject.find_by(oid: '9985f9f77934e73d91dceb46a905393490d29f94475bbc8137e4c266705f7a20').id)
LfsObjectsProject.create(project_id: 337, lfs_object_id: LfsObject.find_by(oid: '00ebf1723c0105e0d69dd64ad260ce211a972e75101a927d320de794d5c30fe9').id)
LfsObjectsProject.create(project_id: 337, lfs_object_id: LfsObject.find_by(oid: '8bcf1836eed411dd367cf7b03404c37d60475a38c58fbb4f76adb2b0a84ababd').id)

etc etc etc… (obviously, with the correct project id / LFS sha256 hashes…)