Local docker runner doesn't restore cache, but shared runners do

I’m trying to compile and test a C++ project and use ccache to speed up my build time.

My .gitlab-ci.yml allows caching just fine with the Gitlab Shared Runners, but when I run the same pipeline with my runner the job reports creating the cache and successfully extracts it on the following run but there are no files in my cache directory.

build:
  stage: build
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - ccache/
  artifacts:
    paths:
      - build/app/wg-ui
      - build/lib/libmetix-lib.so*
      - build/tests/tests
  script:
    - echo "Starting build"
    - ls
    - ls ccache
    - mkdir -p build && cd build
    - qmake ../
    - make -j$(nproc) -Oline CXX="ccache ${CXX}"

Is there a configuration step I’ve missed? Do I need to do anything with my config.toml?