Most people in this situation would probably build their own Docker image, store it in the GitLab container registry, and then use that in their pipeline. This blog post describes how to do that.
If you really want to use caching, that’s only complicated because apt-get puts all its files in odd places. This comment and the rest of that issue should give you some ideas on how to get started.