CI runner caches not being downloaded

GitLab CI/CD
1

I’m having trouble with my CI configuration. It has been working for the past six months or so but it’s suddenly started failing. My setup stage is uploading a cache of node_modules but the subsequent stages are no longer downloading said cache, which causes them to fail. This is on the GitLab.com hosted service.

Here’s my CI config:

cache:
  paths:
    - node_modules/
  key: "$CI_COMMIT_SHA"

image: node:latest

stages:
  - setup
  - test
  - frontend
  - deploy

setup:
  # cache uploaded here (working - "Uploading cache.zip to <url>... Created cache")
  stage: setup
  script:
    - npm install && npx license-checker --exclude 'MIT, ISC, BSD-2-Clause, Unlicense, BSD-3-Clause, Zlib, Apache-2.0, WTFPL, MPL-2.0, CC-BY-4.0, CC-BY-3.0, CC0-1.0, BSD, Public Domain'

test:
  stage: test
  cache:
    # cache is never downloaded in this stage
    policy: pull
  services:
    - mongo:latest
  script:
    - NODE_ENV=test npm run test:lockdown

frontend:
  stage: frontend
  cache:
    policy: pull
  script:
    - npm run bower install --allow-root
    - npm run gulp build

deploy_staging:
  stage: deploy
  environment:
    name: staging
    url: <redacted>
  script:
    - "echo deploy"
  only:
    - master

deploy_prod:
  stage: deploy
  environment:
    name: production
    url: <redacted>
  script:
    - "echo deploying"
  when:
    manual
  only:
    - master

Any help would be greatly appreciated. I’m sure I’ve got something configured wrong but I can’t work out what it is. Many thanks!

2

We have a similar issue on our self hosted gitlab-ce instance. It seems like “$CI_COMMIT_SHA” is not getting evaluated (probably since a gitlab update). I noticed if I use “cache-$CI_COMMIT_SHA” then my logging shows:

Checking cache for cache-…
Successfully extracted cache

I can see $CI_COMMIT_SHA is set as we are using it in a build number later, it just doesn’t work in the key for some weird reason. If you have the same problem we might need to open a bug for it.

3

That’s interesting. No idea why it wouldn’t work in the key.

The whole caching thing was working perfectly until 2017-12-21 10:31pm UTC whereupon all my pipelines started failing. I was told by a member of staff that they cleared their runner cache at 2017-12-22 ~14:00 UTC but the issue continued. What else happened on 21st/22nd December? GitLab 10.3 was released - see the bit under “Strict check on artifacts dependencies”, I think it might be relevent.

I was originally using $CI_BUILD_REF but that was deprecated in GitLab 9.0, and was removed at some point after that. That can’t have helped my problem but I moved to $CI_COMMIT_SHA and I can see that correctly being evaluated in my pipelines now.

However, caches were still 404’ing after changing it so I’ve taken the step of adding npm i to my stages, regardless of whether the cache exists or not. If it exists, it is used and npm won’t re-install, if it doesn’t, npm will go ahead and install the modules.

4

I’m having a similar issue where the cache sometimes does not download - the runner checks for it (correctly evaluating the name, as I’m using variables in the key) but doesn’t find the cache, even though other jobs have found it.

My CI setup first download dependencies (inside a virtualenv and using Pipenv in my case, since I’m using Python), and caches the dependencies, to be picked up by following jobs. Sometimes, retrying the job several times fixes the issue, but with several jobs to restart in a particular order, it just is not feasible manually.