I’m trying to use multiple caches for different directories in my CI/CD pipeline, based on different files. When I run my pipeline, all the different caches all try to use the same key, despite being based on different files.

Env

Running with gitlab-runner 14.8.0~beta.44.g57df0d52 (57df0d52)
  on blue-1.shared.runners-manager.gitlab.com/default j1aLDqxS
Preparing the "docker+machine" executor

Setup

format:
  stage: format
  image: percygrunwald/docker-asdf-terraform-ci-base:v2022-02-10-042739
  cache:
    - key: {files: [.pre-commit-config.yaml]}
      paths: [.pre-commit]
    - key: {files: [requirements-python.txt]}
      paths: [.venv]
    - key: {files: [.tool-versions]}
      paths: [.asdf]
  script:
    - pre-commit run --all-files

Expected behavior

Each of the caches should have a different key. I would presume the key for each of the caches would be a hash of the file itself.

Actual behavior

All 3 caches end up having the same key, which prevents them from pushing and pulling correctly:

Restoring cache
Checking cache for a67e8930d3116b6f4ce940d2f419fe1943c840e0-2...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/a67e8930d3116b6f4ce940d2f419fe1943c840e0-2 
Successfully extracted cache
Checking cache for a67e8930d3116b6f4ce940d2f419fe1943c840e0-2...
cache.zip is up to date                            
Successfully extracted cache
Checking cache for a67e8930d3116b6f4ce940d2f419fe1943c840e0-2...
cache.zip is up to date                            
Successfully extracted cache

As shown above, all 3 caches have the key a67e8930d3116b6f4ce940d2f419fe1943c840e0-2, which is unexpected since each of the cache keys is based on a different list of files.

When updating the caches, the same behavior is seen:

Saving cache for successful job
Creating cache a67e8930d3116b6f4ce940d2f419fe1943c840e0-2...
.pre-commit: found 3367 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/a67e8930d3116b6f4ce940d2f419fe1943c840e0-2 
Created cache
Creating cache a67e8930d3116b6f4ce940d2f419fe1943c840e0-2...
.venv: found 10508 matching files and directories  
Archive is up to date!                             
Created cache
Creating cache a67e8930d3116b6f4ce940d2f419fe1943c840e0-2...
.asdf: found 230 matching files and directories    
Archive is up to date!                             
Created cache
Cleaning up project directory and file based variables
Job succeeded

Once again, all 3 caches appear to have the same key: a67e8930d3116b6f4ce940d2f419fe1943c840e0-2.

I tested this with static keys to confirm the behavior:

format:
  stage: format
  image: percygrunwald/docker-asdf-terraform-ci-base:v2022-02-10-042739
  cache:
    - key: .pre-commit-config.yaml
      paths: [.pre-commit]
    - key: requirements-python.txt
      paths: [.venv]
    - key: .tool-versions
      paths: [.asdf]
  script:
    - pre-commit run --all-files

This had the expected result of having 3 different caches with static keys:

Restoring cache
Checking cache for .pre-commit-config.yaml-2...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/.pre-commit-config.yaml-2 
Successfully extracted cache
Checking cache for requirements-python.txt-2...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/requirements-python.txt-2 
Successfully extracted cache
Checking cache for .tool-versions-2...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/.tool-versions-2 
Successfully extracted cache

Also updates as expected at the end of the pipeline:

Saving cache for successful job
Creating cache .pre-commit-config.yaml-2...
.pre-commit: found 3367 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/.pre-commit-config.yaml-2 
Created cache
Creating cache requirements-python.txt-2...
.venv: found 10284 matching files and directories  
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/requirements-python.txt-2 
Created cache
Creating cache .tool-versions-2...
.asdf: found 230 matching files and directories    
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/32907858/.tool-versions-2 
Created cache
Cleaning up project directory and file based variables
Job succeeded

Does anyone know what’s going on here? I’ve read the caching docs multiple times along with the cache keyword reference and can’t work out why I’m seeing what I’m seeing.

From the docs:

test-job:
  stage: build
  cache:
    - key:
        files:
          - Gemfile.lock
      paths:
        - vendor/ruby
    - key:
        files:
          - yarn.lock
      paths:
        - .yarn-cache/
  script:
    - bundle install --path=vendor
    - yarn install --cache-folder .yarn-cache
    - echo Run tests...

Thanks for any help.

Hey @percygrunwald !
Did you find any solution for this problem? I am facing the same on my GitLab instance.

Hi, I have the same problem and I can solve using prefix

job_name:
  cache:
    - key:
        files:
          - package-lock.json
        prefix: WA

This generate same hash for all files but concatenates the prefix, and it works.
@dev-dsp