GitLab Runner crashes other resources in cluster

Kubernetes cluster goes down when running GitLab Pipeline

In my Kubernetes cluster I am running GitLab-ee 15.8.0 with a GitLab Runner. This runner is configured for a kubernetes executor and I have mounted the /var/run/docker.sock to this runner in the configmap. When running a pipeline which brings up a docker-compose-test.yml, the entire cluster goes down for a few seconds (At least all the Nginx-Ingress stuff, because the DNS isn’t reachable). After that I can see that the pipeline is still in the Running state, but nor runner is working on it. The last command the runner executed in the pipeline was: docker-compose -f docker-compose-test.yml up -d.

  • I expected the pipeline to just bring up the docker containers and run the Laravel tests using the database container and the application container, but instead it messes up the Nginx-Ingress resource.

  • Consider including screenshots, error messages, and/or other helpful visuals

  • What version are you on? Are you using self-managed or GitLab.com?

    • GitLab (Hint: /help): GitLab-ee:15.8.0
    • Runner (Hint: /admin/runners): gitlab/gitlab-runner:latest
  • Here is the gitlab-ci.yml:

image: docker:20.10.16

services:
  - docker:20.10.16-dind

variables:
  DOCKER_COMPOSE_CMD: "docker-compose -f docker-compose-test.yml"
  

stages:
  - test
  - build
  

test:
  stage: test
  script:
    - docker-compose --version
    - $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
    - $DOCKER_COMPOSE_CMD up -d
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test ./scripts/wait-for.sh database-test:54321 -t 60 -- echo "Database connection established"
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test php artisan passport:keys
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test php artisan migrate
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test sh -c "vendor/bin/phpunit ./tests $PARAMETERS --coverage-text --colors=never --stderr"
    - $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
  # only:
  #   - tags

build:
  stage: build
  script:
    - export IMAGE_TAG=$(echo "$CI_COMMIT_TAG" | awk -F '/' '{print $NF}')
    - docker build -t laravel-api:"$IMAGE_TAG" .
    - docker login -u "$CONTAINER_REGISTRY_USERNAME" -p "$CONTAINER_REGISTRY_PASSWORD" "$CONTAINER_REGISTRY_URL"
    - docker push laravel-api:"$IMAGE_TAG"
  only:
    - tags

And this is the docker-compose-test.yml that seems to mess things up:

version: "3.7"
services:
  laravel-api-test:
    build:
      args:
        user: laravel
        uid: 1000
      context: .
      dockerfile: docker/development/Dockerfile
    working_dir: /var/www/
    volumes:
    - ./:/var/www
    ports:
    - ${APP_PORT}:9000
    networks:
    - application

  database-test:
    image: postgres:15.1-alpine
    ports:
    - 54321:5432
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
    networks:
    - application

networks:
  application:
    driver: bridge

The last thing that is probably relevant is the gitlab-runner config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner-config
  namespace: gitlab-runner
data:
  config.toml: |-
    concurrent = 4
    [[runners]]
      name = "Runner_1"
      url = "https://gitlab.project.com/ci"
      token = "my-token"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "gitlab-runner"
        privileged = true
        poll_timeout = 600
        cpu_request = "1"
        service_cpu_request = "200m"
        [[runners.kubernetes.volumes.host_path]]
            name = "docker"
            mount_path = "/var/run/docker.sock"
            host_path = "/var/run/docker.sock"

Finally this the output from the pipeline after it crashed:

Running with gitlab-runner 15.8.2 (4d1ca121)

on Runner_1 eNNz4y9k, system ID: r_y3jEhmF8fN58

Preparing the "kubernetes" executor

00:00

Using Kubernetes namespace: gitlab-runner

Using Kubernetes executor with image docker:20.10.16 ...

Using attach strategy to execute scripts...

Preparing environment

00:04

Waiting for pod gitlab-runner/runner-ennz4y9k-project-117-concurrent-0f24cx to be running, status is Pending

Running on runner-ennz4y9k-project-117-concurrent-0f24cx via gitlab-runner-56cd6f4bb5-zrbd9...

Getting source from Git repository

00:01

Fetching changes with git depth set to 20...

Initialized empty Git repository in /builds/Clients/opus-volvere/laravel-api/.git/

Created fresh repository.

Checking out 3890412c as main...

Skipping Git submodules setup

Executing "step_script" stage of the job script

$ docker-compose --version

Docker Compose version v2.6.0

$ $DOCKER_COMPOSE_CMD down --volumes --remove-orphans

Container laravel-api-database-test-1 Stopping

Container laravel-api-laravel-api-test-1 Stopping

Container laravel-api-database-test-1 Stopping

Container laravel-api-laravel-api-test-1 Stopping

Container laravel-api-database-test-1 Stopped

Container laravel-api-database-test-1 Removing

Container laravel-api-laravel-api-test-1 Stopped

Container laravel-api-laravel-api-test-1 Removing

Container laravel-api-laravel-api-test-1 Removed

Container laravel-api-database-test-1 Removed

Network laravel-api_application Removing

Network laravel-api_application Removed

$ $DOCKER_COMPOSE_CMD up -d

#1 [internal] load build definition from Dockerfile

#1 transferring dockerfile: 827B done

#1 DONE 0.1s

#2 [internal] load .dockerignore

#2 transferring context: 88B done

#2 DONE 0.1s

Please let me know what I am doing wrong here