Disable Gitlab Runner Cache

I am using multiple stages on the gitlab runner. I re-use the cache for npm modules and such. But once it is build i have an artifact and i create a docker image. Another stage deploys the docker image. But even in the stage it will download all the npm modules. Something that takes time and is not needed.

How can i disable that? Disabled all the cache in the runner is not handy.