Kaniko on GitLab shared runner sometimes unusable slow (while unpacking rootfs)

GitLab CI/CD
,
1

Hi,

I use kaniko to build docker images in my GitLab CI Pipeline in a private project on gitlab.com that uses the free shared runners.

For some pipelines I had no problems and everything was fast. But now the jobs are pending long (which is ok, as I’m currently using the free shared runners) but when the jobs are picked up and run, they hang with at this part:

INFO[0003] Unpacking rootfs as cmd RUN rm -rf /opt/docker/etc/nginx/conf.d/10-php.conf requires it.

Are the runners using different filesystems/hardware. It seems unrelated to changes I make in my configuration (e.g. .gitlab-ci.yml or Dockerfile or on which “RUN”-line the unpacking is required as that depends on whether it can use a cache layer or not) it just depends on the runner execution

build:latest-master:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --cache=true --build-arg BUILD_ENVIRONMENT=DEVELOPMENT --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:latest
  only:
    - master

Output of the beginning of the job, (before the mentioned unresponsive/taking too long line):

Running with gitlab-runner 13.1.0 (6214287e)
  on docker-auto-scale ed2dce3a
Preparing the "docker+machine" executor
00:13
Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:390df83ff1abbd538f83962be08373facd5a145f3357466401517728e6a20769 for gcr.io/kaniko-project/executor:debug ...
Preparing environment
00:02
Running on runner-ed2dce3a-project-2061429-concurrent-0 via runner-ed2dce3a-srm-1593505588-c98fbb57...