GitLab Runner fails to get quota size

GitLab CI/CD
,
1

I’m trying to set up GitLab CI/CD for my private instance, but whenever I start a build job, it fails with this error message:

Running with gitlab-runner 16.4.1 (d89a789a)
  on run1 pXbn4uCQS, system ID: s_6374e561057f
Preparing the "docker" executor 00:09
Using Docker executor with image registry.plexor.net/alpine:latest ...
WARNING: Container based cache volumes creation is disabled. Will not create volume for "/cache"
ERROR: Preparation failed: creating cache volume: creating docker volume: Error response from daemon: create runner-pxbn4ucqs-project-1-concurrent-0-7b1a4a19a4b679d7-cache-c33bcaa1fd2c77edfc3893b41966cea8: quota size requested but no quota support (manager.go:203:0s)
Will be retried in 3s ...

My setup is as follows:

  • GitLab running in a dedicated VM (installed using the Linux RPM package).
  • gitlab-runner is running as a systemd service in a separate VM, and is configured to run build jobs in Docker containers.
  • Both GitLab and gitlab-runner are on version 16.4.1, and both are running on fully updated instances of openSUSE Leap 15.5.

There is no mention of quotas in the setup instructions of gitlab-runner, so I’m a bit confused about what the problem is.

Even more confusing, the error occurs even if I disable the cache volume, as shown in the error message above.

Can anyone here shed some light on what’s going wrong?

My gitlab-runner.toml file is as follows:

concurrent = 4
check_interval = 0
shutdown_timeout = 0

[[runners]]
name = "run1"
url = "https://git.plexor.net/"
token = "glrt-pXbn4uCQSeFp7HxscFko"
executor = "docker"
[runners.docker]
  tls_verify = true
  image = "registry.plexor.net/alpine:latest"
  privileged = false
  disable_entrypoint_overwrite = false
  oom_kill_disable = false
  disable_cache = false
  volumes = [
    "/cache",
  ]
  shm_size = 0
  allowed_pull_policies = ["always", "if-not-present"]
  allowed_images = ["registry.plexor.net/*:*"]
  allowed_services = ["registry.plexor.net/*:*"]
  [runners.docker.volume_driver_ops]
    "size" = "50G"
2

Please provide version of Docker (or output of docker info) and type of filesystem on your server.

3 
gitlab-run1:~ # docker info
Client:
 Version:    24.0.5-ce
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.0.0+unknown
    Path:     /usr/lib/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 24.0.5-ce
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 oci runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061b76bb
 init version: 
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.14.21-150500.55.19-default
 Operating System: openSUSE Leap 15.5
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 7.75GiB
 Name: gitlab-run1
 ID: a879023d-8f94-4f15-afd6-df32604baab9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

The filesystem is btrfs.

4

I expected so. Switch to Use the BTRFS storage driver | Docker Docs
In general, I do not recommend running Docker on Btrfs, there have been bunch of issues in the past and I kind of do not trust it anymore.
I think overlay2 is supported on Btrfs, since kernel 5.19 (or something like that), which I am not sure openSuse Leap 15.5 has (or the required changes were backported).

5

Thank you for the hint. I tried to switch to the BTRFS storage driver, but it didn’t help.

Then I reinstalled the OS for the runner, this time with ext4 as the file system, and I recreated the setup of the runner (except both gitlab and gitlab-runner are at version 16.5.0 now). But the same error still occurs whenever I try to launch a build job:

Running with gitlab-runner 16.5.0 (853330f9)
  on run1 LuVdWFQSi, system ID: s_99f568c3612f
Preparing the "docker" executor 00:12
Using Docker executor with image registry.plexor.net/alpine:latest ...
Using helper image:  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-853330f9
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-853330f9 ...
Using docker image sha256:dcb973b518cf1c49d1ec7c327ba7e1a0978cfee5845d75ad10e45ef72b79a3e2 for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-853330f9 with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:06704ac91ed7f81c38d679b7863f31610072d6c8a9741a14b7ef9333ab5638be ...
ERROR: Preparation failed: adding cache volume: creating docker volume: Error response from daemon: create runner-luvdwfqsi-project-1-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70: quota size requested but no quota support (manager.go:203:0s)
Will be retried in 3s ...

Now docker info produces the following output:

gitlab-run1:~ # docker info
Client:
 Version:    24.0.5-ce
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.0.0+unknown
    Path:     /usr/lib/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.5-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 oci runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8c087663b0233f6e6e2f4515cee61d49f14746a8
 runc version: v1.1.9-0-gccaecfcbc907
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.14.21-150500.55.31-default
 Operating System: openSUSE Leap 15.5
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 7.75GiB
 Name: gitlab-run1
 ID: 0574fd6e-eff3-42f0-b3df-caafd29311fe
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Any suggestions on how to proceed?

6

Try to remove this from the config.toml

1 Like
7

That was it - removing those lines fixed the issue.

The configuration I used comes from your documentation, so perhaps you have to amend it, or at least explain what the removed part does and point out it’s optional: Docker executor | GitLab

Thanks a lot for your help.

1 Like
8

OK… but there was a benefit of this option, right? To limit too large docker volumes within the GitLab workflow. Which storage drivers are all supported? Or is it about the backing filesystem?

Could you provide an easy way to migrate to that other filesystem or storage driver (ideally not btrf).

But I fully agree, please remove this section from the example. It’s explain further down in the documentation as optional.