Gitlab Auto DevOps build fails with the error: `cgroups: cgroup mountpoint does not exist: unknown`

Using the Nodejs Express Template, with Auto DevOps, the build pipeline fails with the following error:

Step 6/10 : RUN npm install
 ---> Running in b6e0eef8227c
cgroups: cgroup mountpoint does not exist: unknown

This was not an issue before I did an update to my cluster.

  • cri-o 1.19 to 1.20.
  • linux 5.11.6 to 5.11.15.

A Rollback to before the cluster update, fixes the issue.

Don’t know if this is GitLab related or not.
May be a docker bug related to newer kernel or container runtime.

Setup

  • openSUSE kubic
  • linux 5.11.15-1-default
  • GitLab 13.10.3
  • gitlab-runner 13.9.0
  • kubernetes 1.20.2
  • cri-o 1.20.2

I solved the issue, and was able to get unprivileged gitlab-runner working on CRI-O.

I edited the Auto DevOps .gitlab-ci.yml template file, removed the default build and added a custom build that uses kaniko.

I followed the guide on gitlab docs and tweaked it to get it working with my setup.
https://docs.gitlab.com/ee/ci/docker/using_kaniko.html

Here is a snippet for future reference, when eventually builds start breaking on newer kernels/cri-o versions for people.

.gitlab-ci.yml

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - |
      if [[ -z "$CI_COMMIT_TAG" ]]; then
        export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG}
        export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_SHA}
      else
        export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE}
        export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_TAG}
      fi

      image_tagged="$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
      image_latest="$CI_APPLICATION_REPOSITORY:latest"
      
      mkdir -p /kaniko/.docker
      echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
      
      /kaniko/executor \
        --force \
        --context $CI_PROJECT_DIR \
        --dockerfile $CI_PROJECT_DIR/Dockerfile \
        $AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS \
        --destination "$image_tagged" \
        --destination "$image_latest"
  rules:
    - if: '$AUTO_DEVOPS_PLATFORM_TARGET == "EC2"'
      when: never
    - if: '$CI_COMMIT_TAG || $CI_COMMIT_BRANCH'