Why git lfs binary files are converted to pointers during git checkout?

GitLab CI/CD
, , ,
1

:hugs: Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Why git lfs binary files are converted to pointers during git checkout?

Describe your question in as much detail as possible:

  • What are you seeing, and how does that differ from what you expect to see?
  • Consider including screenshots, error messages, and/or other helpful visuals

In our ci job, the GIT_LFS_SKIP_SMUDGE is set to 0 because we need to use the newest git lfs binary files. We want to speed up the CI job and the idea is to use the pvc mounted to each k8s job pod for consistent storage, to be more specific, to reuse the git repo and the bazel cache. However, the git lfs pull process is too slow(around 5min) :frowning: . It is unexpected because we suppose the ci job can use the git repo instead of pull everything each time.
To debug this problem, I did a lot of work:
I added some git configs to the runner to trace the git operation and performance like the following:
GIT_TRACE: 1
GIT_TRACE_PERFORMANCE: 1
GIT_TRACE_SETUP: 1
Meanwhile, I also added “ls -la /path” logs before and after the critical operation (git fetch/git checkout/git clean) in the source and build the binaries.

image
image1920×1054 185 KB

image
image1920×950 241 KB

image
image1920×914 185 KB

You can see the binaries in the pvc are converted to pointers during check out process! This is the reason why the following git lfs pull process is slow.

image
image1920×887 166 KB

The question is that why git lfs binary files are converted to pointers during git checkout? Anyone can help me out??? Thank you!!!

Steps to reproduce

  • create pvc/pv resource
  • use pvc working for consistent storage for our project git repo and bazel cache.
  • deploy the k8s runner by helm chart
  • trigger a pipeline and the job is executed by the pvc runner

Configuration

Add the CI/CD configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml). Alternatively, create a public GitLab.com example project that provides all necessary files to reproduce the question.

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • [ this one ] Self-hosted Runners

Versions

Helpful resources

  1. Before opening a new topic, make sure to search for keywords in the forum search
  2. Check the GitLab Runner and GitLab projects for existing issues. If you encounter a bug, please create a bug report issue.
  3. Troubleshooting docs: GitLab Runner, self-managed GitLab instances.

Thanks for taking the time to be thorough in your request, it really helps! :blush: