Using cache for pip/npm dependencies in Gitlab CI

Hi,

We’ve been using Gitlab.com (not self-managed) for the last few weeks. We want to use the shared runners to execute our CI, and I succeed to set up a config with our existing suite tests.

The main stage passes, however it takes about 22 minutes compared to 10-12 minutes on our legacy CI for one main reasons : Pypi and npm packages are downloaded and re-installed/compiled at each pipeline, which takes minutes (definitely most of the 10 extra minutes, maybe the whole).

Our .gitlab-ci.yml looks like this right now. Sorry for the long paste, but I prefer to give as much context as possible:

image: "python:3.7-alpine"

variables:
  [... some db/tokens variables...]
  # Set pip's cache inside the project directory since we can only cache local items
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

stages:
  - test
  - coverage

cache:
  key: pip-and-npm-global-cache
  paths:
    - $CI_PROJECT_DIR/.cache/pip
    - $CI_PROJECT_DIR/.cache/npm

before_script:
  - mkdir -p $CI_PROJECT_DIR/.cache/pip $CI_PROJECT_DIR/.cache/npm

django_tests:
  stage: test

  services:
    - postgres:9.6-alpine
    - mongo:3.6-xenial

  cache:
    key: "coverage-$CI_COMMIT_REF_SLUG"
    paths:
      - .coverage

  script:
    # Various packages required to run dependencies below
    - apk add [...]
    - pip install pip --upgrade
    - pip install -r requirements.txt
    - coverage ...  # execute tests here

js_tests:
  stage: test
  image: "node:alpine"
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - node_modules/

  script:
    - npm ci --cache $CI_PROJECT_DIR/.cache/npm --prefer-offline
    - npm install && npm run build
    - npm run test

coverage:
  stage: coverage
  script:
    - pip install coverage==4.5.3 django_coverage_plugin==1.6.0
    - coverage report -i -m [...]

First, the “test” stage always re-installs the packages, even between two builds on the same branch without any new commits. The stage passes though (as said before) but the coverage one doesn’t because some pip requirements installed before are not available anymore.
I have the same problem with a local runner on my machine and with the shared runners of Gitlab.com.

I tried to set some ls in the script and it seems $CI_PROJECT_DIR/.cache is always empty at the start of a job (django_tests and coverage). Did I miss something? Does any of my cache declaration overlaps another one?

2 Likes

I allow myself to resurrect my question. Does anyone have already encounter this problem? Does anyone has a config example working for pip?

3 Likes

Same thing here. I searched every corner of the internet for someone who had some working example or application. I have a successful case with maven, but for python it’s a different template and i have tried everything within my knowledge but without any success.

Hi @mlorant and @luciojb,

the Python example from the official documentation has worked pretty well for me in the past:
https://docs.gitlab.com/ee/ci/caching/#caching-python-dependencies

My understanding of the cache configuration is that job-level directives override global ones, so each of your jobs seems to be using a different cache, and only the last job is using the global one. If all jobs are supposed to use the same cache, try using only a global cache configuration and see if that helps.

One thing I would like to point out about the Python example above is that it caches the venv directory it installs packages to in addition to the Pip package cache. This should prevent jobs from re-installing the same packages every time.

Kind regards,
Alexander

1 Like

Hi @mlorant,
I used to have the very same problem up until a couple of weeks ago, when I finally found a webpage setting it up in the right way.
First of all, you have to keep in mind, that when a job creates a cache, even with the broadest matching key, it will only stay local to the runner it was created on, with a bit of luck a future job might pull it. Watch out, default policy is pull-push, so every job can rewrite the cache and potentially wipe content you want to preserve. In the case of “pip-and-npm-global-cache”, the npm and python job are trashing each others content. Most important thing I did, was get a Minio container and provide a shared cache. Next I change policy to pull and had the first job take care of filling the pip cache and installing venv. The whole thing works best though, when venv is saved as artifact!

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  XDG_CACHE_HOME: "$CI_PROJECT_DIR/.cache"

image: python:3.8-slim-buster

## These paths will be cached in between test runs. Saving the download times.
cache: &package_cache
  key:
    files:
      - poetry.lock
    prefix: poetry
  paths:
    - .cache
  policy: pull

stages:
  - Prepare
  - Static Analysis

before_script:
  - apt-get update &&
      apt-get install -qqy --no-install-recommends --no-install-suggests
        make
  - pip install poetry

dependencies:
  stage: Prepare
  cache:
    <<: *package_cache
    policy: pull-push
  script:
    - python -m venv --copies venv
    - source venv/bin/activate
    - python -m pip install --upgrade pip
    - poetry export --without-hashes -n |
        tee requirements.txt
    - poetry export --dev --without-hashes -n |
        tee requirements-dev.txt |
        pip install -r /dev/stdin
    # pip cache filled with all package downloads
    - poetry build
  artifacts:
    paths:
      - requirements.txt
      - requirements-dev.txt
      - dist/
      - .venv/
    exclude:
      - .venv/**/__pycache__/*
    when: on_success
    # venv gets extracted for each job and is immutable

# The uncompromising Python code formatter
black:
  stage: Static Analysis
  script:
    - make lint-test

# Simple and scalable tests for Python code
pytest:
  stage: Static Analysis
  script:
    - poetry install
    - make test
  artifacts:
    when: on_success
    reports:
      cobertura: coverage.xml
      junit: report.xml

Found the webpage again, that gave me the idea: GitLab CI: Cache and Artifacts explained by example

I hope that helps

1 Like