CI/CD Cache - Weird behavior with `needs`?

Context

My goal is to create a CI “job cache” for a composer, but only the “dev” part,
not the global composer for a production environment.
This will allow me to add some testing during the CI with require-dev vendors.

My thought process is to:

1. Download the dev vendors (composer install).
2. Push them into the cache.
3. In other jobs, pull this cache.
4. Use the vendor cache.

Config

Here’s my gitlab-ci.yml:

variables:
    GIT_STRATEGY: fetch
    GIT_SUBMODULE_STRATEGY: recursive
    GIT_SUBMODULE_DEPTH: 1
    GIT_DEPTH: "5"
    GET_SOURCES_ATTEMPTS: "3"
    RESTORE_CACHE_ATTEMPTS: "3"
    ARTIFACT_DOWNLOAD_ATTEMPTS: "3"
    FF_USE_FASTZIP: "true"
    ARTIFACT_COMPRESSION_LEVEL: "fast"
    CACHE_SHARED: "true"
    CACHE_COMPRESSION_LEVEL: "fast"
    SENTRY_LOG_LEVEL: info
    DOCKER_DRIVER: overlay

default:
    image: allsoftware/symfony:5-php-8.1-v10
    interruptible: true
    tags:
        - allsoftware-deployment

stages:
    - setup
    - build
    - tests

.vendor-dev-cache: &vendor_dev_cache
    key:
        files:
            - composer.lock
        prefix: dev
    paths:
        - vendor/
    policy: pull

composer_dev_install:
    stage: setup
    cache:
        -   <<: *vendor_dev_cache
            policy: pull-push
    needs: [ ]
    variables:
        APP_ENV: dev
        APP_DEBUG: 1
    script:
        - cp .env.gitlab-runner .env.local
        - composer install --no-scripts --no-interaction --prefer-dist --optimize-autoloader

phpstan:
    stage: tests
    cache:
        -   <<: *vendor_dev_cache
    needs:
        - composer_dev_install
    variables:
        APP_ENV: dev
        APP_DEBUG: 1
    script:
        - ls -lah vendor # <---- crash HERE
        - php bin/console cache:clear --no-warmup
        - php -d memory_limit=4G vendor/bin/phpstan analyse -c phpstan.neon

translations_lint:
    stage: tests
    cache:
        -   <<: *vendor_dev_cache
    needs:
        - composer_dev_install
    variables:
        APP_ENV: dev
        APP_DEBUG: 1
    script:
        - ls -lah vendor
        - php bin/console lint:yaml translations
        - php bin/console lint:xliff translations

Error

When using this config, phpstan job doesn’t get the vendor/ directory, and fail on the ls command.

image1326×856 49.1 KB

OR the translation_lint has succeed!

image1641×2411 447 KB

Weird behavior

Switching job places between phpstan and translations_lint allows the phpstan job to run but fail on the translations_lint.

# [...]
.vendor-dev-cache: &vendor_dev_cache
    key:
        files:
            - composer.lock
        prefix: dev
    paths:
        - vendor/
    policy: pull

composer_dev_install:
    stage: setup
    cache:
        -   <<: *vendor_dev_cache
            policy: pull-push
    needs: [ ]
    variables:
        APP_ENV: dev
        APP_DEBUG: 1
    script:
        - cp .env.gitlab-runner .env.local
        - composer install --no-scripts --no-interaction --prefer-dist --optimize-autoloader

translations_lint:
    stage: tests
    cache:
        -   <<: *vendor_dev_cache
    needs:
        - composer_dev_install
    variables:
        APP_ENV: dev
        APP_DEBUG: 1
    script:
        - ls -lah vendor # <--- Now crash HERE
        - php bin/console lint:yaml translations
        - php bin/console lint:xliff translations

phpstan:
    stage: tests
    cache:
        -   <<: *vendor_dev_cache
    needs:
        - composer_dev_install
    variables:
        APP_ENV: dev
        APP_DEBUG: 1
    script:
        - ls -lah vendor
        - php bin/console cache:clear --no-warmup
        - php -d memory_limit=4G vendor/bin/phpstan analyse -c phpstan.neon

What is the problem???

I assumed that it comes from needs behavior, so I ended up putting phpstan IN the needs of translations_lint, but it doesn’t help.

From:

To:

image856×78 8.89 KB

Or the inverse:

image828×98 4.45 KB

Version

• GitLab: 16.1.2-ee
• Runner: 16.1.0 (865283c5)

Hi,

I will start with this note from the official docs: Caching is an optimization, but it isn’t guaranteed to always work. You might need to regenerate cached files in each job that needs them.

back to the problem, issues with cache are mostly caused by GitLab Runner configuration, rather then Pipeline misconfig. Please share your GitLab Runner config.toml and describe setup of your GitLab Runner.

Hi, thanks for your answer.

Here is the config.toml

concurrent = 5
check_interval = 0
​
[session_server]
  session_timeout = 1800
​
[[runners]]
  name = "gitlab-ci-xxxxxxxxx"
  url = "https://gitlab.xxxxxxxx.xx/"
  token = "xxxxxxxxxxx"
  executor = "docker"
  [runners.cache]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    memory = "2048m"
#    memory_swap = "256m"
    memory_reservation = "256m"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

For the setup, nothing touchy:

• Gitlab self-hosted
• On a debian (OVH public cloud)
• Using gitlab-runner package

You are missing cache_dir under [[runners]]. Try to add it and restart the Runner to see if it helps:

concurrent = 5
check_interval = 0
​
[session_server]
  session_timeout = 1800
​
[[runners]]
  name = "gitlab-ci-xxxxxxxxx"
  url = "https://gitlab.xxxxxxxx.xx/"
  token = "xxxxxxxxxxx"
  executor = "docker"
  cache_dir = "/cache"
  [runners.docker]
...