Trouble Moving LFS folder to another mounted disk

How to Use GitLab
,
1

Versions

Server: Self Managed GitLab CE 15.11.3 (Linux Package)
OS: Ubuntu 20.04.6 LTS

Background

Hello, I am hosting my own GitLab server on a computer with a small main disk (512GB) and a massive RAID volume (8TB). The idea was to have the server installed on the main disk and to have the repo data live on the RAID volume. Before my issue I was able to successfully move the git-data and backups folders to my RAID volume, but I didn’t realize that LFS content was still on my main disk.

Because of this, just recently my main disk ran out of space when a user pushed a ton of LFS files to the server. The solution to me was to move the lfs-objects folder to my RAID volume as well, but since doing that I’ve been having issues with some (but not all) LFS operations.

Here are the steps I followed to do this:

  • Stop the gitlab Instance and move the LFS dir
sudo gitlab-ctl stop
rsync -av --delete <old-dir> <new-RAID-dir>
  • Update the config file. Added this line to the gitlab.rb
gitlab_rails['lfs_storage_path'] = "<new-RAID-dir>"
  • Reconfigure and restart the server
sudo gitlab-ctl reconfigure
sudo gitlab-ctl start

I should note that I couldn’t really find any official docs that told me to do it this way, and followed some instructions from here that in a similar manner move the git-data directory.

Resulting Issues

I can make a new repository and push/pull new LFS objects just fine, but when cloning a repo that contains old LFS objects from before the directory move, I get the following error client-side:

git checkout dev
2024/05/08 06:43:21 protocol error: received DATA after END_STREAM
Downloading Assets/Art/Bee/Bee_Avatar.fbx (886 KB)MiB/s
Error downloading object: Assets/Art/Bee/Bee_Avatar.fbx (f684c15): Smudge error: Error downloading Assets/Art/Bee/Bee_Avatar.fbx (f684c155744d8bccb9d5ac2f49ba25921777adbfa2e68642a08f386b67827e4b): expected OID f684c155744d8bccb9d5ac2f49ba25921777adbfa2e68642a08f386b67827e4b, got e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 after 0 bytes written

Errors logged to 'C:<REPO-DIR>\.git\lfs\logs\20240508T064335.2529467.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: Assets/Art/Bee/Bee_Avatar.fbx: smudge filter lfs failed

And looking at the generated log this is the relevant blurb at the top (Can include more if needed)

git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)
git version 2.42.0.windows.2

$ git-lfs.exe filter-process
2024/05/08 06:43:21 protocol error: received DATA after END_STREAM
Error downloading object: Assets/Art/Bee/Bee_Avatar.fbx (f684c15): Smudge error: Error downloading Assets/Art/Bee/Bee_Avatar.fbx (f684c155744d8bccb9d5ac2f49ba25921777adbfa2e68642a08f386b67827e4b): expected OID f684c155744d8bccb9d5ac2f49ba25921777adbfa2e68642a08f386b67827e4b, got e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 after 0 bytes written

expected OID f684c155744d8bccb9d5ac2f49ba25921777adbfa2e68642a08f386b67827e4b, got e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 after 0 bytes written
...

Troubleshooting

I found this post that seems nearly identical to my issue. I tried the recommended steps as follows, but no beans.

  • Disable directory management in gitlab.rb and reconfigure
    manage_storage_directories[‘enable’] = false
  • Make sure new dir is writable
    chmod -R 770 lfs-storage-directory
  • Make sure new dir is owned by git user (also tried git:git)
    chown -R git:root lfs-storage-directory
    Also to clarify, the git-data and backups directories were both owned by “git:root”, so I think permissions are correct here.

I also found this post from 2019 which never got any answers :cry:. This is still definitely relevant to me.

Next Steps

Still troubleshooting and trying to figure out where to go from here, any advice is very appreciated! At the very least I have a backup of the server from this weekend so I have a fallback if I really need it.

  • Figure out why new LFS objects are fine and old LFS objects are inaccessible. It doesn’t seem like the new LFS objects are going to a new/different directory, but that might be the key to solving this
  • Create a minimum reproduction on another machine.

Additionally, it might make more sense to just backup, uninstall, and reinstall the entire server on the RAID disk. The whole point of that disk is to preserve our files if the main disk conks out, but since the non-data part of the server live on the main disk anyway we’d still have to use a backup if the main disk died. I haven’t done anything like this before and could appreciate advice about this as well.