We’ve been using Gitlab.com (not self-managed) for the last few weeks. We want to use the shared runners to execute our CI, and I succeed to set up a config with our existing suite tests.
The main stage passes, however it takes about 22 minutes compared to 10-12 minutes on our legacy CI for one main reasons : Pypi and npm packages are downloaded and re-installed/compiled at each pipeline, which takes minutes (definitely most of the 10 extra minutes, maybe the whole).
Our .gitlab-ci.yml looks like this right now. Sorry for the long paste, but I prefer to give as much context as possible:
image: "python:3.7-alpine" variables: [... some db/tokens variables...] # Set pip's cache inside the project directory since we can only cache local items PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" stages: - test - coverage cache: key: pip-and-npm-global-cache paths: - $CI_PROJECT_DIR/.cache/pip - $CI_PROJECT_DIR/.cache/npm before_script: - mkdir -p $CI_PROJECT_DIR/.cache/pip $CI_PROJECT_DIR/.cache/npm django_tests: stage: test services: - postgres:9.6-alpine - mongo:3.6-xenial cache: key: "coverage-$CI_COMMIT_REF_SLUG" paths: - .coverage script: # Various packages required to run dependencies below - apk add [...] - pip install pip --upgrade - pip install -r requirements.txt - coverage ... # execute tests here js_tests: stage: test image: "node:alpine" cache: key: "$CI_COMMIT_REF_SLUG" paths: - node_modules/ script: - npm ci --cache $CI_PROJECT_DIR/.cache/npm --prefer-offline - npm install && npm run build - npm run test coverage: stage: coverage script: - pip install coverage==4.5.3 django_coverage_plugin==1.6.0 - coverage report -i -m [...]
First, the “test” stage always re-installs the packages, even between two builds on the same branch without any new commits. The stage passes though (as said before) but the coverage one doesn’t because some pip requirements installed before are not available anymore.
I have the same problem with a local runner on my machine and with the shared runners of Gitlab.com.
I tried to set some
ls in the script and it seems
$CI_PROJECT_DIR/.cache is always empty at the start of a job (django_tests and coverage). Did I miss something? Does any of my cache declaration overlaps another one?