Fresh self-managed GitLab with LFS does not work?

I am having trouble with LFS on a fresh install of GitLab EE 16.10.1. Here is what I’m doing:

  1. Create an empty project on the GitLab server

  2. Add it as a remote to a local git repository with LFS-tracked files

  3. git push --all to the new remote

    I get an error:

    ...
    remote: GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".
    To http://server/group/repo.git
     ! [remote rejected]   main -> main (pre-receive hook declined)
    ...
    
  4. git lfs push --all

    Success; I see that the LFS files are uploaded in the GitLab repository’s Usage Quotas section.

  5. git push --all again

    I get the same error:

    ...
    remote: GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".
    To http://server/group/repo.git
     ! [remote rejected]   main -> main (pre-receive hook declined)
    ...
    

It’s worth noting that the same steps 1-3 work without issue on an instance of GitHub Enterprise Server (I am comparing the two).

I haven’t knowingly changed any of the default settings on my GitLab server; LFS seems to be enabled by default.

My understanding is that there are no hooks set up by default, so I’m a bit perplexed by the pre-receive hook error itself. When I look around in the server, I can’t find any hook definitions.

I’ve read through the GitLab LFS docs, the GitLab LFS administration docs, and the GitLab LFS troubleshooting docs with no luck.

I’ve also found a couple of seemingly/potentially related issues (below), but they don’t seem to have recommended steps to resolve. It feels like I must be missing something obvious?

I learned that LFS adds hooks when you add it to a repository, which is how it works; seems kind of obvious now, looking back, but it doesn’t really explain the behavior I am seeing.

I’m still not sure what the underlying problem is, but I’ve found a workaround for what I need for now:

  1. Push only the LFS files with git lfs push --all
  2. Disable LFS on the repository in the GitLab settings
  3. Push the non-LFS files with git push --all --no-verify
  4. Re-enable LFS on the repository in the GitLab settings

Once these steps are complete, the repository seems to be working as expected; new LFS files seem like they can be added/tracked and pushed without issue. The LFS files (new and old) show up as LFS files in the GUI and show the correct LFS size.

I did try to make a simplified repository to better demonstrate this issue, but I couldn’t quickly recreate it; I was able to git push --all from the beginning and get the expected behavior.
A few variations I tried:

  • LFS files on one branch that have always been tracked
  • LFS files on multiple branches that have always been tracked
  • LFS files on multiple branches that were migrated later

I think this narrows the issue down to a couple of possibilities that I didn’t cover:

  • the size of the files being tracked
  • the number of files being tracked
  • the length of the repository history

There’s definitely something going on, but with the workaround described above I’m no longer blocked. I’m still interested in what’s happening if anyone has insights.