Isolate parallel jobs build dir using gitlab-runner docker executor

Problem to solve

We are using a gitlab-runner with docker executor.

Very often, when two or more jobs run in parallel in our pipeline, the two jobs “step on each other”, one succeeds and the other fails with this:

The documentation says that Docker Executor “Maintain the same build environment for each job.”. I’m not sure if that’s related but it seems to me that if “each job” have the “same build environment”, they share it.

What we want is a complete isolation between jobs, no matter what they are doing. We do not share files or cache or anything using a build dir.

We also tried to use a custom build dir, using various values that we thought would be unique for each job, but they turn to have the same value so no hope with this.

What should we do here? Thanks

Steps to reproduce

Setup a machine with Docker.
Create a runner group, get a token and register a new runner that way:

docker run -d \
            --name "gitlab-runner" \
            --volume /var/run/docker.sock:/var/run/docker.sock \
            --restart always \
            "gitlab/gitlab-runner:alpine3.18-v16.6.1"
docker exec "gitlab-runner" gitlab-runner register \
            --non-interactive \
            --url "https://gitlab.com" \
            --token "${gitlab-registration-token}" \
            --executor "docker" \
            --docker-image "alpine:3.18" \
            --docker-volumes "/var/run/docker.sock:/var/run/docker.sock" \
            --description "${environment} (docker)"
docker restart "gitlab-runner"
docker exec "gitlab-runner" gitlab-runner run

This launches a runner in docker, and it will use the host’s docker daemon to run jobs with the defined image.

Configuration

In your project, have a CI such as this one (adapt your tags to the runner you’ve registered…):

stages:
  - run-parallel

run-one:
  stage: run-parallel
  environment:
    name: staging
  image:
    name: debian:latest
  script:
    - hostname
    - ls -li

run-two:
  stage: run-parallel
  environment:
    name: staging
  image:
    name: debian:latest
  script:
    - hostname
    - ls -li

Versions

Versions

  • GitLab GitLab Enterprise Edition 17.4.0-pre de710e6b02e
  • GitLab Runner 16.6.1

It seems that the dedicated variables to handle concurrent jobs are not enough.

Instead relying on the job ID actually provides uniqueness:

variables:
  GIT_CLONE_PATH: $CI_BUILDS_DIR/project-$CI_PROJECT_ID-job-$CI_JOB_ID

Then i don’t understand the purpose of CI_CONCURRENT_ID and CI_CONCURRENT_PROJECT_ID variables, which stay at zero no matter how much jobs are running on the same runner.