Caching dependencies in Gitlab CI using docker-compose and docker in docker (dind)

GitLab CI/CD
1

I am running a slef-hosted gitlab instance. For my pipelines I am using docker-in-docker (dind). I build my container and run my tests on it.

Recently I added pandas as a dependency and now it takes ages to run the pipelines. I would like to somehow cache my builds so I don’t have to build the docker container everytime anew. The problem is that I am reading alot that I can’t do that with docker-in-docker. Is that really so or can I somehow adjust either my config.toml or my gitlab-ci.yml to cache the dependencies?

My gitlab-ci.yml looks like this:

image: docker:19.03.0

variables:
  DOCKER_DRIVER: overlay2
  # Create the certificates inside this directory for both the server
  # and client. The certificates used by the client will be created in
  # /certs/client so we only need to share this directory with the
  # volume mount in `config.toml`.
  DOCKER_TLS_CERTDIR: "/certs"

services:
  - docker:19.03.0-dind

stages:
  - unittests
  - publish-doc

tests-website:
  stage: unittests
  before_script:
    - apk update
    - apk upgrade
    - apk add python python-dev py-pip build-base libffi-dev openssl-dev libgcc
    - pip install docker-compose
  script:
    - docker-compose -f test-headless.yml build
    - docker-compose -f test-headless.yml up --no-start
    - docker-compose -f test-headless.yml start selenium
    - docker-compose -f test-headless.yml run django pytest
  tags:
    - docker

Help on how to set this up would be very much appreciated. Thanks in advance!!

2

Belated reply but hope it helps.

I think it’s possible to use the DinD approach while still caching images. The trick is to have GitLab deploy the DinD service container with a host volume bind-mounted into the DinD container’s /var/lib/docker directory. This way, any images downloaded by the Docker daemon running inside the DinD container will be “cached” at host level, and subsequent instances of the DinD container will have access to that cache.

Try adding a /var/lib/docker to the volumes clause in the GitLab runner’s config.toml:

    [runners.docker]
        ...
        volumes = ["/certs/client", "/cache", "/var/lib/docker"

But note: since you are caching, this means the DinD container no longer starts with a clean Docker cache. Thus, your jobs need to explicitly cleanup any state that you don’t desire to persist across jobs.

Also, you may want to check out this blog post I wrote on GitLab + Docker:

There are plenty of diagrams there, plus a solution on how to avoid security / isolation problems associated with DinD. The blog also talks about caching.

Hope that helps!