Dynamically create a CI job if cache is not available or valid anymore

GitLab CI/CD
1

Hi,

I use the CI caching system to cache my project dependencies. In the case of JavaScript I only recreate the cache inside a job if my yarn.lock file changes. This caches the node_modules folder and, in general, runs pretty smoothly for all my pipelines.

Unfortunately, it can happen that the cache gets deleted, and because yarn.lock didn’t change, it also does not recreate the cache. So, all following jobs that depend on the cache will fail.

Can I somehow also start the cache-creation job, if the current cache does not exist? (I just found a manual work around, but this is not my preferred solution. I also look for an environment variable, but didn’t found one.)

default:
  # https://docs.gitlab.com/ee/ci/caching/
  cache: &global_cache
    key:
      files:
        # only generate a new cache if this files changes
        # same cache over multiple pipelines
        - yarn.lock
    paths:
      - node_modules
    policy: pull

install_dependencies_in_cache:
  stage: .pre
  cache:
    # inherit all global cache settings
    <<: *global_cache
    # override the policy
    policy: pull-push
  script:
    - yarn install --frozen-lockfile
  rules:
    # only if dependencies changed, refill the cache
    - changes:
        - yarn.lock
    # TODO: trigger if cache does not exist - how?
    # ...
    # work-around: trigger manually if cache does not exist
    - when: manual
      allow_failure: true

Thank you

2 Likes
2

I think one important takeaway from Gitlab’s documentation is

Caching is an optimization, but isn’t guaranteed to always work, so you need to be prepared to regenerate any cached files in each job that needs them.

Is there a big reason why the install_dependencies_in_cache job can’t run everytime? If the dependencies are already in the cache, when you run yarn install, it should take minimal time since all of the dependencies should already exist.

3

thanks for the answer. Yes, always running the job could also be a solution (but not directly what I want).

Actually, I wanted to optimize the CI pipeline time, and one of the slowest activities in my setup is loading and writing the cache. (So, not yarn install is the problem.) That’s why I use the pull policy by default, and that’s also why only the caching job has the pull-push policy enabled.

2 Likes