Caching dependencies in Gitlab CI using docker-compose and docker in docker (dind)

I am running a slef-hosted gitlab instance. For my pipelines I am using docker-in-docker (dind). I build my container and run my tests on it.

Recently I added pandas as a dependency and now it takes ages to run the pipelines. I would like to somehow cache my builds so I don’t have to build the docker container everytime anew. The problem is that I am reading alot that I can’t do that with docker-in-docker. Is that really so or can I somehow adjust either my config.toml or my gitlab-ci.yml to cache the dependencies?

My gitlab-ci.yml looks like this:

image: docker:19.03.0

variables:
  DOCKER_DRIVER: overlay2
  # Create the certificates inside this directory for both the server
  # and client. The certificates used by the client will be created in
  # /certs/client so we only need to share this directory with the
  # volume mount in `config.toml`.
  DOCKER_TLS_CERTDIR: "/certs"

services:
  - docker:19.03.0-dind

stages:
  - unittests
  - publish-doc

tests-website:
  stage: unittests
  before_script:
    - apk update
    - apk upgrade
    - apk add python python-dev py-pip build-base libffi-dev openssl-dev libgcc
    - pip install docker-compose
  script:
    - docker-compose -f test-headless.yml build
    - docker-compose -f test-headless.yml up --no-start
    - docker-compose -f test-headless.yml start selenium
    - docker-compose -f test-headless.yml run django pytest
  tags:
    - docker

Help on how to set this up would be very much appreciated. Thanks in advance!!