I have created my own gitlab-ci.yml for my project. The project is fairly big (about 2GB) and has lots of different projects in a single repo.
I created 2-3 stages in my pipeline, it creates python virtualenv, downloads pip’s modules, and build Rust package. I used cache to store pip’s downloaded packages.
BUT, I have 3 pip local modules that are being installed in my repo (I mean the package location is within the repo) so I added those directories to my cache as well.
Question 1: Is this ok? I mean if I cache my own repo’s some directories, it will always replace with the cached ones even if I pushed a new code in those?
Question 2: AFAIK, in every run and every time the pipeline starts, the cache restores the last archive right? I mean If a pipeline finishes, the runner holds the cache and uses that cache every time or the caches are being removed after each pipeline?
Question 3: Is there an anyway better way to improve the whole process? I need the pip downloaded packages in every stage.
variables: PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" cache: paths: - .cache/pip - venv/ - my_dir1/ - my_dir2/ stages: - install_req - refresh_pack - cargo before_script: - python -V - pip install virtualenv - virtualenv venv - source venv/bin/activate install_req: stage: install_req script: - pip install --cache-dir=$PIP_CACHE_DIR psycopg2 - cd $CI_PROJECT_DIR/my_dir1/ - pip install --cache-dir=$PIP_CACHE_DIR -r requirements.txt - cd $CI_PROJECT_DIR/my_dir2/ - pip install --cache-dir=$PIP_CACHE_DIR -r requirements.txt refresh_pack: stage: refresh_pack script: - rustup toolchain install nightly cargo: stage: cargo script: - cargo check