Gitlab ci - prevent job getting stuck if no runner with tag is available

GitLab CI/CD
,
1

I have a pipeline with multiple jobs that can run independently. But there is one “init” job, that needs to run on the same VM before the other jobs! For redundancy reasons I set up the CI config, such that the jobs can run on different VMs. My goal here is, that if one of the VMs fail, the pipeline still runs without problems. I managed to set this up with the following configuration:

For this example lets say I have 2 VMs, each VM has a runner with the following tags:

  • VM1:
    • tags: [vm1, vms]
  • VM2:
    • tags: [vm2, vms]

My gitlab.ci config looks like this:

stages:
  - init
  - run

init_job:
  stage: init
  parallel:
    matrix:
      - TYPE: vm
        TAGS: [1, 2]
  tags:
    - ${TYPE}${TAGS}
  allow_failure: true
  retry: 0
  timeout: 10m
  script:
    - echo "INIT JOB"

job1:
  stage: run
  tags:
    - vms
  script:
    - echo "JOB 1"

job2:
  stage: run
  tags:
    - vms
  script:
    - echo "JOB 2"

job3:
  stage: run
  tags:
    - vms
  script:
    - echo "JOB 3"

If both runners are up, everything works fine. But if eg the runner of VM1 is not availabe (because VM1 is down), the “init_job” with tag “vm1” gets stuck in pending state forever (the timeout unfortunately does not apply for pending time).

So here are my questions:

  • Is there a way to configure the CI such that either the init_jobs fails or gets canceled if no runner is available?
  • Can I set the “waiting time” somewhere?
  • Is there another way to set up the “init_job” to make sure its running on each VM?
  • Is there a better way to get this kind of VM redundancy?

A appreciate any help!
Thanks :slight_smile:

2

Why don’t you just use

tags:
  - vms

on the init_job like for other jobs? In this case it will be picked up by either runner on VM1 or VM2.

3

the init job needs to run on both VMs

4
  • Is there a way to configure the CI such that either the init_jobs fails or gets canceled if no runner is available?

No

  • Can I set the “waiting time” somewhere?

No

  • Is there another way to set up the “init_job” to make sure its running on each VM?

No

  • Is there a better way to get this kind of VM redundancy?

No, jobs should be independent from the location they are running in.

If you need some steps to be executed, add them to the job* jobs like before_script or directly to script.