Edit: I found that my problem is exactly the same as described in this open issue: Backend: short-circuit CI pipelines based on if cache exists (#224650) · Issues · GitLab.org / GitLab · GitLab
Problem to solve
I have the following job in my .gitlab-ci.yml. It installs the dependencies of my project and caches them, so that jobs in later stages can extract the cache and lint and test the project. For each branch, an individual cache is created using CI_COMMIT_REF_SLUG, as different branches may in principle have different (new) dependencies.
poetry:
stage: deps
rules:
- changes:
- poetry.lock
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- .cache/pip
- .poetry
- .venv
policy: pull-push
before_script:
- echo Installing dependencies ...
script:
- python3 -m venv $POETRY_HOME
- $POETRY_HOME/bin/pip install poetry==1.8.3
- $POETRY_HOME/bin/poetry config virtualenvs.in-project true
- $POETRY_HOME/bin/poetry install
When the file poetry.lock changes, that means that the dependencies changed and the cache needs to be recreated. This situation is covered by the job rules and works for branch pipelines when new commits are pushed. It also works for branch pipelines when new branches are pushed, because then rules: changed is always true.
However, the job is not added to a new merge request pipeline for a merge request that does not modify poetry.lock, because then the file is compared to its version in the target branch. Because dependencies are cached per-branch, this means that subsequent jobs fail (unless the project is configured to run branch pipelines in addition to merge request pipelines, which I would like to avoid).
Every once in a while, I have to delete old caches on my runner to reclaim disk space. In that case, branches or merge requests that had working pipelines before start failing their pipelines, as the dependency job has no way to know that the cache is missing.
How can I add my dependency job to all branch and merge request pipelines if the cache is missing or a cahnge in poetry.lock demands reinstallation of the dependencies?
Steps to reproduce
Put the job above in a .gitlab-ci.yml that is configured for merge request pipelines only and create a new merge request. The job will not be added to the pipeline and there will be no dependencies for subsequent jobs.
Put the job above in a .gitlab-ci.yml that is configured for branch pipelines only and push a new branch. The job will be added to the pipeline initially and create the cache. Delete the cache and push a new commit to the branch. The cache will not be recreated.
Configuration
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
# - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
# when: never
# - if: $CI_COMMIT_BRANCH
variables:
PIP_CACHE_DIR: $CI_PROJECT_DIR/.cache/pip
POETRY_HOME: $CI_PROJECT_DIR/.poetry
default:
image: python:3.11
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- .cache/pip
- .poetry
- .venv
policy: pull
before_script:
- source $($POETRY_HOME/bin/poetry env info --path)/bin/activate
stages:
- deps
- lint
- test
poetry:
stage: deps
rules:
- changes:
- poetry.lock
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- .cache/pip
- .poetry
- .venv
policy: pull-push
before_script:
- echo Installing dependencies ...
script:
- python3 -m venv $POETRY_HOME
- $POETRY_HOME/bin/pip install poetry==1.8.3
- $POETRY_HOME/bin/poetry config virtualenvs.in-project true
- $POETRY_HOME/bin/poetry install
pre-commit:
stage: lint
script:
- source .gitlab-ci/pre-commit.sh
pytest:
stage: test
script:
- pytest
Versions
Please select whether options apply, and add the version information.
- Self-managed
- GitLab.com SaaS
- Self-hosted Runners