Problem to solve
I am running a self hosted distributed gitlab setup which currently (as per default behavior) has LFS enabled and stores LFS objects on the gitlab rails servers. Periodically a backup is run which should include these objects and stored away.
Due to an incident we had to restore from one of these backups, and after restoring it was noticed that a substantial number of LFS objects were missing and checking the servers it indeed appears to be the case. I then checked a number of the backups taken and it is not a one off, they all have this file missing in the lfs dir. It appears that this file was simply never backed up at all despite being on the server and despite being a part of the repo.
As far as I know the gitlab backup utility should backup all lfs objects and put them into the archive. Is there any way for me to still recover those objects? If not how do I stop this situation from happening again?
Steps to reproduce
- Take backup with
gitlab-backup create - Restore the backup later with
gitlab-backup restore
Configuration
Gitlab-ee deployed with gitlab omnibus
Versions
Please select whether options apply, and add the version information.
- Self-managed
Versions
- GitLab: 15.10