How to cache node_modules globally for all pipelines for a project

How to cache node_modules for all pipelines in a project

I would like to avoid running the yarn install part if the yarn.lock file has not changed for all subsequent pipeline runs for the project. For this I would need to cache the node_modules and some how to NOT run the yarn install part if the yarn.lock file hasn’t changed. If anyone can please help and explain how to set up the cache globally? Below (after the snippet)i have included another snippet of what my interpretation of how to configure the cache would look like after reading some of the documentation.

Here is my current snippet of my .gitlab-ci.yml*

stages:
  - bootstrap
  - package:deploy

.install-lint-test: &install-lint-test
    - yarn install
    # --quiet hides all the warnings
    - npx eslint --quiet './packages/**/*.js' './packages/**/*.jsx'
    - npx eslint --quiet './widgets/**/*.js' './widgets/**/*.jsx'
    - npx eslint --quiet './components/**/*.js' './components/**/*.jsx'
    - npx eslint --quiet './tools/**/*.js' './tools/**/*.jsx'
    # tests
    - npm run test:ci

bootstrap:feat:
  stage: bootstrap
  image: rbuenavida360/node10-chromeheadless-alpine:latest
  script:
    - *install-lint-test
  only:
    - branches
  artifacts:
    paths:
      - dist/
  except:
    - tags
    - master
    - development
    - integration

bootstrap:dev:
  stage: bootstrap
  image: rbuenavida360/node10-chromeheadless-alpine:latest
  extends:
    - .build_cache
  script:
    - *install-lint-test
    # Build assets
    - npx lerna exec -- npm version prerelease --preid=${CI_PIPELINE_IID}
    - echo "--> Build scripts and packages"
    - npx lerna run --concurrency 4 build -- --env=dev & npx lerna run --concurrency 4 build -- --env=prod
    - npm run pack:all
    - npm run copy:all -- -- -- --skipDocs --env=dev --env=prod
  only:
    - development
  except:
    - tags
  artifacts:
    expire_in: 2h
    paths:
      - dist/
...etc

Based on what i’m interpreting, the cache should be defined before the stages part like so:

cache:
  key:
    files:
      - yarn.lock
  paths:
    - node_modules/
    - "packages/**/node_modules/"
    - "components/**/node_modules/"
    - "widgets/**/node_modules/"
    - "tools/**/node_modules/"
  policy: pull

1 - Is this correct?
2 - I need help defining the missing part in my gitlab ci to run the yarn install and push to the global cache only if the yarn.lock file has changed.

Any help would be greatly appreciated :blush:

Hi @rbuenavida

Unfortunately the docs here use npm, and I haven’t done this myself. However, Yarn has some docs here that include a GitLab config, and there’s a forum answer that looks relevant. Between the two I think that might cover most of what you’re looking for?

Sarah

Ok, thank you. I’m now speaking with my devops team regarding a message i saw in the pipeline

checking cache for b1a33fd400114f77ee43c884262fccfec8583ce8...e
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.

So i’m checking with devops to see what this means. Unless maybe you might have an idea ?

Hi there! I think this means that you are using a local cache on your server and not, say an S3 AWS server.

You can check your runner configs, and there is some more information in the docs here: https://docs.gitlab.com/ee/ci/caching/#where-the-caches-are-stored

Yes, thank you. That makes sense. There is a config for the cache for the runners.

@ rbuenavida

I do have the same issue. how did you solved your issue ? What was your final config looks like ?

Updated docs have information on yarn: Caching in GitLab CI/CD | GitLab