[self-hosted runners] How to get Gitlab Cache to work with Golang

GitLab CI/CD
, ,
1

Problem to solve

This is all happening on my own self-hosted runner, setup using NixOS (options)

I am trying to make sure all the jobs running CI use a cache, rather than needing to keep downloading the go module in each job.

I tried to follow the example in the docs here.

However, with my CI configuration, I notice some jobs always seem to use the cache and others don’t. Even when their config looks identical (besides the script part).

For example, the lint job uses the cache

lint:
  extends:
    - .task
  script:
    - task lint

Link to job: lint (#7942729201) · Jobs · Haseeb Majid / Banter Bus · GitLab

but in another job, I see the following warning:

Checking cache for default-non_protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
WARNING: Cache file does not exist                 
Failed to extract cache

generate:sqlc:
  extends:
    - .task
  script:
    - task generate:sqlc
    - git diff --exit-code

Link to job: generate:sqlc (#7942729206) · Jobs · Haseeb Majid / Banter Bus · GitLab

Steps to reproduce

  • Create a new branch
  • Push some changes
  • Create a new MR
  • Go to pipelines and view jobs
  • Some jobs fail to pull cache
  • Others always pull the cache

You can see here we failed to pull the cache and then also download the go deps again.

image
image1294×1140 185 KB

Some example pipelines: fix: broken ci cache (!38) · Merge requests · Haseeb Majid / Banter Bus · GitLab

Configuration

Here is a simplified version of my .gitlab-ci.yml file:

image: nixos/nix
workflow:
  auto_cancel:
    on_new_commit: interruptible
default:
  interruptible: true
variables:
  IMAGE_TAG: 0.1.0
  BANTERBUS_CI_IMAGE: $CI_REGISTRY_IMAGE/ci:$IMAGE_TAG
stages:
  - pre
  - deps
  - test
  - build
  - release
.task:
  stage: test
  image:
    name: $BANTERBUS_CI_IMAGE
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
  cache:
    paths:
      - ${GOPATH}/pkg/mod
    policy: pull
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  before_script:
    - mkdir -p .go
    - task build:dev
download:dependency:
  extends: .task
  stage: deps
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - go mod download
  cache:
    policy: pull-push
lint:
  extends:
    - .task
  script:
    - task lint
format:
  extends:
    - .task
  script:
    - task format
generate:sqlc:
  extends:
    - .task
  script:
    - task generate:sqlc
    - git diff --exit-code

Link to full config: .gitlab-ci.yml · ef13833eb4774ea99fab77e5f4533544fe1633ef · Haseeb Majid / Banter Bus · GitLab

My GitLab runner config in Nix: modules/nixos/services/gitlab-runner/default.nix · 06dd3d4f68337264a38e3a3d61a50ffb77c0bf03 · Haseeb Majid / nixicle · GitLab

and then on the machine itself sudo cat /var/lib/gitlab-runner/.gitlab-runner/config.toml

concurrent = 10
[[runners]]
name = "default_ms01_1d713ad183f0"
limit = 10
url = "https://gitlab.com"
# redacted token config
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "debian:stable"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
network_mtu = 0

Versions

  • Self-hosted Runners
gitlab-runner --version
Version:      17.2.0
Git revision: v17.2.0
Git branch:   HEAD
GO version:   go1.22.7
Built:        unknown
OS/Arch:      linux/amd64

Please let me know if you need any more information. Any help would be greatly appreciated, thanks.

2

I think it’s important to note a few things here-

  1. I assume this is just one runner. If not, you need a distributed cache (a la S3 like Minio or actual S3, Google, etc.)

  2. The cache is per project and per protected branch. So different projects will not pick up the same cache as they have different docker volumes that will not mix. Additionally, each protected branch will have a different cache as a suffix is added to each cache key. See- Caching in GitLab CI/CD | GitLab

So if it is the last point- you can go into your CI/CD settings and clear that checkbox for separate caches and it should work for any branches on that project. The rest at a quick glance looks good to me.