Is there a way to use OR condition with needs in GitLab CI job

I am trying to create a job dependency with “OR” condition for previous stage jobs using “needs” in “.gitlab.ci.yml” file but unable to find a solution for this.

.gitlab-ci.yml file →

stages:
  - build
  - test
  - deploy


Build_job:      
  stage: build
  script:
    - echo "hello from build job"

Test_job1:
  stage: test
  script:
    - echo "Start test 1"
  when: manual

Test_job2:
  stage: test
  script:
    - echo "Start test 2"
  when: manual

Deploy_job:
  stage: deploy
  script:
    - echo "Start deploying the job"
  when: manual
    needs:
      - job: Test_job1
        optional: true
      - job: Test_job2
        optional: true

My aim is either of Test_job1 or Test_job2 is passed, Deploy_job should be enabled. But with the above code, I am unable to do so as Deploy_job is getting enabled only when both previous two test jobs are passed.

Is there a way if something can be used like needs: [Test_job1 or Test_job2] ?

3 Likes

I am also looking for a solution for this, did you come up with one?
Thanks

I did some very intensive research on this subject a few days ago (documentation, forum articles, the “whole” web), but found no solution.

I “solved” (okay, it’s a very ugly workaround!) my CI/CD pipeline by duplicating all related jobs and assigning them as single followers of the needed job with the OR condition.

The price tag for this workaround: the later jobs might be executed multiple times per pipeline (which might be unexpected, but acceptable behaviour sometimes, but sometimes it might be not acceptable, too?!).

Feature request

pre-build-job1:
  script:
    - echo "Prepare some stuff 1, but might fail"

build-job1:
  needs: pre-build-job1
  script:
    - echo "Some stuff 1"

build-job2:
  when: manual
  script:
    - echo "Some stuff 2"

post-build-job-with-or-condition:
  needs: (build-job1 OR build-job2)  # <= this line is NO VALID syntax and just demonstrates my requirement
  script:
    - echo "Do some stuff if job 1 or job 2 were successful

job-test:
  needs: post-build-job-with-or-condition:
  script:
    - echo "Some testing"

job-deploy:
  needs: job-test
  script:
    - echo "Deployment"

Very ugly workaround

pre-build-job1:
  script:
    - echo "Prepare some stuff 1, but might fail"

build-job1:
  needs: pre-build-job1
  script:
    - echo "Some stuff 1"

build-job2:
  when: manual
  script:
    - echo "Some stuff 2"


# Base implementation for jobs pipeline for build-job1 OR build-job2
post-build-job-with-or-condition-base:
  script:
    - echo "Do some stuff if job 1 or job 2 were successful

job-test-base:
  script:
    - echo "Some testing"

job-deploy-base:
  script:
    - echo "Deployment"


# Jobs pipeline for build-job1
post-build-job1-dependent:
  needs: build-job1
  extends: post-build-job-with-or-condition-base

job1-test:
  needs: job1-dependent
  extends: job-test-base

job1-deploy:
  needs: job1-test
  extends: job-deploy-base


# Jobs pipeline for build-job2
post-build-job2-dependent:
  needs: build-job2
  extends: post-build-job-with-or-condition-base

job2-test:
  needs: job2-dependent
  extends: job-test-base

job2-deploy:
  needs: job2-test
  extends: job-deploy-base

Same question here. We would like to have an “OR” condition for using “needs” or to have the possibility to set an “at least one” flag for the array of needs.

1 Like

Same question here. In our case the use-case is a manual deploy job to one of three UAT environments. We would like to implement the “needs” relationship that deployment to one of the three UAT environments needs to have been successful for a production deployment to be allowed.

TL;DR; since it appears the conditional logic is not supported by needs, the solution I see for your case is to move Deploy_job to another pipeline that gets triggered by successful run of Test_job1 or Test_job2. The new pipeline would need to handle the case of both successful and “throttle” so that only the first one in is effective. See reference.

I came here from a similar but different need - to apply a condition to a GitLab CI Pipeline job’s needs - and so far I don’t see how it’s currently possible, since the documentation describes needs as a Job-level-only keyword which does not support any conditional parameter (like when).

In my case, I’ve got early build/push docker image stage/job that only runs when docker image dependencies change, i.e.:

docker-build-and-push:
  stage: update-images
  tags:
    - shell
  rules:
    - changes:
      - Dockerfile*
      - .dockerignore
  script:
    - make docker-build docker-push

and a later stage which must be run afterwards, non-concurrently:

make-eks:
  stage: create-management-cluster
  tags:
    - shell
  # needs:
  #   job: docker-build-and-push
  resource_group: pipeline_mutex
  script:
    - make eks

So I would like to uncomment that needs clause and of course this doesn’t work, syntactically nor functionally. The needs condition is needed because of the docker-build-and-push stage/job’s condition (the rules).

What I’ve arrived at that’s acceptable for now is to rely on GitLab CI stages’ normal sequential ordering, as described in Basic Pipelines. This works in my case because I have the luxury of being able to define the dependency as a single stage/job. Your situation seems a little different.

Sorry to not be offering a complete solution (yet), but maybe we can continue discussion and figure it out.

1 Like

there is a good solution available:

we can use optional for each job which we list under needs

  needs:
    - job: docker-build-VITE
      optional: true
    - job: docker-build-WEBPACK
      optional: true

a complete example:

docker-build-VITE:
  extends: .docker
  needs: ['docker-clean']
  rules:
    - if: $FRONTEND_TOOLING == "VITE"
  script:
    - export VITE_API_URL=$API_URL
    - docker build --build-arg=VITE_API_URL --build-arg=NPM_PKG_AUTH_TOKEN --build-arg=FUNCTIONAL_USER_TOKEN --target production --tag $DOCKER_URL/$PROJ_GROUP_ID/$PROJ_ARTIFACT_ID:$PROJ_VERSION -f Dockerfile .

docker-build-WEBPACK:
  extends: .docker
  needs: ['docker-clean']
  rules:
    - if: $FRONTEND_TOOLING == "WEBPACK"
  script:
    - docker build --build-arg=NPM_PKG_AUTH_TOKEN --build-arg=FUNCTIONAL_USER_TOKEN --target production --tag $DOCKER_URL/$PROJ_GROUP_ID/$PROJ_ARTIFACT_ID:$PROJ_VERSION -f Dockerfile .

docker-login:
  extends: .docker
  needs:
    - job: docker-build-VITE
      optional: true
    - job: docker-build-WEBPACK
      optional: true
  script:
    - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD $DOCKER_URL
6 Likes

I came here with the same question and this is exactly the solution I needed.

Thanks!

In my point of view, it would be sufficient if we could have an hybrid mode stage / dag :

Apart from the DAG feature that comes with needs, needs also provides the dotenv artifcat feature that is very convenient.
In addition this dotenv feature is often related to switching between develop / main branch.

The DAG would be built per stage and we could still rely on stage to define a preprocessing specific to develop / main branch.

Please find below a simplified example that is currently not working, but would be solved if I could use a per stage DAG instead of a per pipeline DAG:

variables:
  CLUSTER_NAME: ""
  TENANT_ENV: ""

stages:
  - pre
  - install
  
.create_dotenv:
  stage: pre
  script:
    - echo ${CI_PIPELINE_SOURCE}
    - echo "IKS_CLUSTER_NAME=${CLUSTER_NAME}" >> build.env
  artifacts:
    paths:
      - build.env
    reports:
      dotenv: build.env

create_dotenv:
  extends: .create_dotenv
  only:
    - tags
    - main

create_dotenv_test:
  extends: .create_dotenv
  before_script:
    - source .gitlab-ci/test.env
  except:
    - tags
    - main

install:
  stage: install
  script:
    - ./bootstrap.sh
  # IMPOSSIBLE WITH GITLAB
  # needs:
  #   - job: create_dotenv OR create_dotenv_test
  #     artifacts: true

and the testing variables .gitlab-ci/test.env are defined as :

# only export variable if it is undefined
export_if_void()
{
    key=`echo $1 | awk -F '=' '{print $1}'`
    if [[ -z ${!key} ]]; then
        export $1
    fi
}
export_if_void CLUSTER_NAME=my-test-cluster
export_if_void TENANT_ENV=test

This enables me to have a pipeline that fails on its own on main or tags because I want it to be only triggered by other pipelines.
However, in test branches it is ok to use a set of predefined variables pointing to test cluster.

I just saw your hint about the magic feature of needs : “optional: true” … thanks @tobiashochguertel
It solved my case above !

just needs:

install:
  stage: install
  script:
    - ./bootstrap.sh
  needs:
    - job: create_dotenv
      artifacts: true
      optional: true
    - job: create_dotenv_test
      artifacts: true
      optional: true

There’s a problem for enabling

optional: true

option for all needed jobs.

That last job that we want to run after EITHER previous jobs
also runs when any of previous jobs are NOT running,
even if stage is set correctly.

I solved something like this by using the Gitlab API to cancel one job when the other is run. I have a similar script that runs the other job with a “bypass: true”. when that flag is set the job runs but does nothing. I added the bypass job so I could control the final pipeline status.

#!/bin/sh
#set -x

# Usage: abort_job <job name>
# Environment:
#   CI_PROJECT_ID - ID number of the project where the job is running
#   CI_PIPELINE_ID - ID number of the pipeline running in the project
#   GL_ACCESS_TOKEN - Token with API access to modify the pipeline


this_script=$0
job_name=$1

# log <LEVEL> <Message>
log() {
  lvl=$1
  msg=$2
  echo "$1:$this_script:`date`:$2"
}


if [ -z "$job_name" ]
then
  log "ERROR" "Missing Parameter, Usage: $0 <job_name>"
  exit 1
fi

log "INFO" "Cancel Job $job_name"
PROJ_API="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID"
PIPE_API="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID"

job_id=`curl --request GET "$PIPE_API/jobs?scope[]=manual"  \
             --header "PRIVATE-TOKEN: $GL_ACCESS_TOKEN" | \
       jq --arg job_name "$job_name"  -c '.[] | select(.name == $job_name) | .id'`


if [ -z "$job_id" ]
then
  log "ERROR" "Unable to find job ID for $job_name"
  exit 1
fi

log "INFO" "Job $job_name has ID: $job_id"

curl --request POST "$PROJ_API/jobs/$job_id/cancel" \
     --header "PRIVATE-TOKEN: $GL_ACCESS_TOKEN" | jq

log "INFO" "Job $job_name has been canceled"
exit 0