Run Gitlab Runner Kubernetes as non root with Kaniko

I set up the gitlab-runner kubernetes executor to run with a non root user “gitlab-runner”

Problem to solve

i get it working for all images except for Kaniko image , i build a custom kaniko image from the official kaniko image and i put the new user owner of the folder “/kaniko” .
But when i run my pipeline i see that all the folder and subfolder of “/” is owner by root inclusing my “/kaniko”
Wht i get override the folders of my image ? any idea

Configuration

image:
  registry: registry.gitlab.com
  image: gitlab-org/gitlab-runner
rbac:
  create: true
  rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
  clusterWideAccess: true
serviceAccountName: gitlab-runner
  podSecurityPolicy:
    enabled: false
    resourceNames:
      - gitlab-runner
runners:
  config: |
    [[runners]]
      builds_dir = "/tmp/builds"
      environment = ["HOME=/tmp"]
      [runners.kubernetes]
        namespace = "{{ item.namespace }}"
        image = "private-registry.internal.custom.com/custom-gitlab-runner-jdk11:latest"
        tls_verify = false
        poll_timeout = 3600
        output_limit = 50000000
        allowed_pull_policies = ["always", "if-not-present"]
        [runners.kubernetes.pod_security_context]
          run_as_non_root = true
          run_as_user = 10000
          run_as_group = 10000
          fs_group = 10000
        [runners.kubernetes.init_permissions_container_security_context]
          run_as_user = 10000
          run_as_group = 10000
      [runners.kubernetes.node_selector]
        "type"="runner"
      [runners.kubernetes.node_tolerations]
        "type=runner" = "NoSchedule"
      [runners.cache]
        Type = "s3"
        Path = "runner"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "oos.eu-west-2.outscale.com"
          BucketName = "{{ item.bucket_name | default("k8s-gitlab-runner-cache") }}"
          BucketLocation = "eu-west-2"
          Insecure = false
          AuthenticationType = "access-key"
  name: "{{ item.name }}"
  privileged: true
  secret: "{{ item.secret_name }}"
  cache:
    secretName: s3access
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  runAsNonRoot: true
  privileged: false
  capabilities:
    drop: ["ALL"]
podSecurityContext:
  runAsUser: 100
  fsGroup: 65533
  supplementalGroups: [10000]
volumes: []

Gitlab-ci

build-job:
  stage: build-image-with-kaniko
  image:
    name: private-registry.internal.custom.com/gitlab-runner-kaniko:1.1.60-SNAPSHOT
    pull_policy: always      
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"private-registry.custom.internal.com\":{\"auth\":\"$(printf "%s:%s" "${REGISTRY_USER}" "${REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor 
      --context . 
      --dockerfile Dockerfile
      --destination private-registry.internal.custom.com/dpd-ubuntu:latest
  variables:
  tags:
    - gitlab-runner-rootless

Capture1222×778 103 KB

Versions

• • Self-managed with Kubernetes executor : gitlab-runner 16.3.3

Thank you for your help

U need to setup you’r volumes to bind with runner config.toml

I send u my config might be help:

runners:
    locked: false
    secret: "gitlab-runner-secret"
    config: |
      [[runners]]
        [runners.kubernetes]
          image = "ubuntu:22.04"
          helper_image = "registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:x86_64-v16.11.0"
          [runners.kubernetes.pod_security_context]
            run_as_non_root = true
            run_as_user = 999
            run_as_group = 999
            fs_group = 999
          [[runners.kubernetes.volumes.empty_dir]]
            name = "kaniko-docker"
            mount_path = "/kaniko/.docker/"
            medium = "Memory"
          [[runners.kubernetes.volumes.secret]]
            name = "gitlab-ca-secret"
            mount_path = "/kaniko/ssl/certs/"
            read_only = true
            [runners.kubernetes.volumes.secret.items]
              "ca.crt" = "/kaniko/ssl/certs/ca.crt"
        [runners.cache]
          Type = "s3"
          Path = "gitlab-runner"
          Shared = true
          [runners.cache.s3]
            ServerAddress = "minio.domain.local"
            BucketName = "runner-cache"
            BucketLocation = "us-east-1"
            Insecure = false
  podAnnotations:
    gitlab.com/prometheus_scrape: "true"
    gitlab.com/prometheus_port: 9252
  podSecurityContext:
    runAsUser: 999
    fsGroup: 999
  volumeMounts:
    - name: root-gitlab-runner
      mountPath: /.gitlab-runner
  volumes:
    - name: root-gitlab-runner
      emptyDir:
        medium: "Memory"

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    # entrypoint: [""]
  script:
    - AUTH=$(echo -n "${DOCKER_USERNAME}:${DOCKER_PASSWORD}" | base64)
    - |
      cat << EOF > /kaniko/.docker/config.json
      {
          "auths": {
              "docker-push.nexus.domain.local": {
                  "auth": "${AUTH}"
              }
          }
      }
      EOF
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
  rules:
    - if: $CI_COMMIT_TAG