Runner cannot mount cache volume

GitLab CI/CD
, ,
1

Problem to solve

Gitlab-runner is failing preparation on any job for not being able to mount the cache volume to /gitlab-runner-cache-init. Checking the ownership and permissions along the path /home/gitlab-runner/.local/share/containers/storage/volumes/runner-axv-aeyxc-project-8-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data showed ownership and group to be gitlab-runner:gitlab-runner with 0700 permissions on each folder except _data and /home/gitlab-runner, which are 0755.

For build, the alpine:3.20 image is used with our build environment added, but the error shows up using any image (hello-world, in this case).

Running with gitlab-runner 17.2.0 (6428c288)
  on Runner_1 axV-AeyxC, system ID: s_3b47d5af9fed
Resolving secrets
Preparing the "docker" executor
00:14
Using Docker executor with image hello-world ...
ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "619319c65549ae9200340bc052bd88b56dec4eb2b33220d29cd3840a37f1cdb2" for volume "runner-axv-aeyxc-project-8-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8": starting permission container: Error response from daemon: runc: runc create failed: unable to start container process: error during container init: error mounting "/home/gitlab-runner/.local/share/containers/storage/volumes/runner-axv-aeyxc-project-8-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data" to rootfs at "/gitlab-runner-cache-init": mount /home/gitlab-runner/.local/share/containers/storage/volumes/runner-axv-aeyxc-project-8-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data:/gitlab-runner-cache-init (via /proc/self/fd/6), flags: 0x5026: operation not permitted: OCI permission denied (linux_set.go:105:0s)
Will be retried in 3s ...
...repeat of error above 3 more times...

Steps to reproduce

If I run podman outside gitlab-runner (as user gitlab-runner), using the following runs as expected with the volume (which is empty).

 podman run -it --volume /home/gitlab-runner/.local/share/containers/storage/volumes/runner-uh3xe6fs9-project-8-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data:/gitlab-runner-cache-init:rw,z alpine:3.20 bash

Configuration

gitlab - self managed
gitlab-runner - self managed

The gitlab-runner file /etc/gitlab-runner/config.toml:

# @file config.toml
#
concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0
log_level = "info"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Runner_1"
  url = "https://pts12gitlab.example.com"
  id = 3
  token = "<TOKEN>"
  token_obtained_at = 2024-08-20T12:22:08Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  environment = ["GIT_SSL_NO_VERIFY=true"]
  tls-skip-verify = true
 [runners.docker]
   pull_policy = "if-not-present"
   host = "unix:///run/user/980/podman/podman.sock"
   tls_verify = false
   image = "alpine-3.20_dev"
   privileged = false
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

The gitlab-ci yaml file:

# @file .gitlab-ci.yml

default:
  image: hello-world

stages:          # List of stages for jobs, and their order of execution
  - test
  - document

unit-test-job:   # This job runs in the test stage.
  stage: test    # It only starts when the job in the build stage completes successfully.
  script:
    - echo "Running unit tests."
    - make test
    - make coverage
    - echo "Unit tests complete."

lint-test-job:   # This job also runs in the test stage.
  stage: test    # It can run at the same time as unit-test-job (in parallel).
  script:
    - echo "Linting code."
    - make lint
    - echo "Linting complete."

document-job:      # This job runs in the document stage.
  stage: document  # It only runs when *both* jobs in the test stage complete successfully.
  script:
    - echo "Making Doxygen report"
    - make doxygen
    - echo "Doxygen report PDF generated."

The output of podman info:

host:
  arch: amd64
  buildahVersion: 1.33.12
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.module+el8.10.0+23498+f7d19d48.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 185e18f210690b795e20f9b83cffa02cc3c90e3d'
  cpuUtilization:
    idlePercent: 82.09
    systemPercent: 0.78
    userPercent: 17.13
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "8.10"
  eventLogger: file
  freeLocks: 2043
  hostname: pst12gitlab
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 980
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 980
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
  kernel: 4.18.0-553.82.1.el8_10.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 7534940160
  memTotal: 33705009152
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.9.4-23.module+el8.10.0+23498+f7d19d48.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.4.0-dev
        commit: unknown
        CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
    package: containernetworking-plugins-1.4.0-6.module+el8.10.0+23498+f7d19d48.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.12-6.module+el8.10.0+23498+f7d19d48.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.2.0+dev
      go: go1.24.6 (Red Hat 1.24.6-1.module+el8.10.0+23407+428597c7)
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/980/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.module+el8.10.0+23498+f7d19d48.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 420052992
  swapTotal: 1048571904
  uptime: 174h 27m 25.00s (Approximately 7.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/gitlab-runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/gitlab-runner/.local/share/containers/storage
  graphRootAllocated: 47179505664
  graphRootUsed: 17624989696
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 41
  runRoot: /tmp/containers-user-980/containers
  transientStore: false
  volumePath: /home/gitlab-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1757584016
  BuiltTime: Thu Sep 11 05:46:56 2025
  GitCommit: ""
  GoVersion: go1.24.6 (Red Hat 1.24.6-1.module+el8.10.0+23407+428597c7)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated
  • Self-hosted Runners

Versions

  • GitLab (Web: /help or self-managed system information sudo gitlab-rake gitlab:env:info):
  • GitLab Runner, if self-hosted (Web /admin/runners or CLI gitlab-runner --version):

Helpful resources

  1. Check the FAQ for helpful documentation, issues/bugs/feature proposals, and troubleshooting tips.
  2. Before opening a new topic, make sure to search for keywords in the forum search
  3. Check the GitLab and GitLab Runner projects for existing issues. If you encounter a bug, please create a bug report issue.
  4. Review existing troubleshooting docs.

Thanks for taking the time to be thorough in your request, it really helps! :blush: