Gitlab Selfhosted: LFS files not uploaded after losing stroage


I have an interesting issue with Gitlab LFS integration.

I enabled LFS and added a S3 (Contabo S3 compatible) storage as a backend for it.

Unfortunately, I forgot to enable the proxy download of data in the config. So I did it, but then I ended up on 404 Errors. Then I tried to re-upload the data (so it will ge uploaded with the right config and work just fine), that did not help, so I got a little bit angry and I recreated the storage completely with deleting all the data there (as it was costing me money and was not working) and I wanted to reupload the data from my local machine, as I still have the one LFS repos on my laptop with all the data unfortunately, that also did not work, as somehow Gitlab still thinks all the data is there and does not upload it.

I googled, how to remove the data from the remote of LFS and I found that I need to deleted all the repos using that data, so I did (it was the first commit and only one repo) while keeping all the data locally.

Then I created new repo (with few new LFS files/images) and tried to push it. It pushed and all the new data was pushed and is accessible correctly via Gitlab, but all the files that were in the first batch with wrong config still ends up on 404.

SO I want to ask for help, how to reupload of fix those files, so I can access them from the LFS.

OK, small update:
It looks like its random what is being shown correctly and what will throw 404.

This is how it looks with working image:

And here is page with file in the same folder not working:

and when you click to download it, it will throw the 404 posted in the first post.

I also thought about the filename lenght, but thats not it either:

I also tried to migrate from the Object Storage to the local storage with LFS, but it does not work with this error:

E, [2022-09-28T17:16:04.396129 #2000030] ERROR -- : Failed to transfer LFS object 65077e0a94a78248db1eb6555fdb22de37efc4ed4fc91660a6db23b002bd22cd with error: undefined method `body' for nil:NilClass