I am trying to create a job dependency with “OR” condition for previous stage jobs using “needs” in “.gitlab.ci.yml” file but unable to find a solution for this.
.gitlab-ci.yml file →
stages:
- build
- test
- deploy
Build_job:
stage: build
script:
- echo "hello from build job"
Test_job1:
stage: test
script:
- echo "Start test 1"
when: manual
Test_job2:
stage: test
script:
- echo "Start test 2"
when: manual
Deploy_job:
stage: deploy
script:
- echo "Start deploying the job"
when: manual
needs:
- job: Test_job1
optional: true
- job: Test_job2
optional: true
My aim is either of Test_job1 or Test_job2 is passed, Deploy_job should be enabled. But with the above code, I am unable to do so as Deploy_job is getting enabled only when both previous two test jobs are passed.
Is there a way if something can be used like needs: [Test_job1 or Test_job2] ?
2 Likes
I am also looking for a solution for this, did you come up with one?
Thanks
I did some very intensive research on this subject a few days ago (documentation, forum articles, the “whole” web), but found no solution.
I “solved” (okay, it’s a very ugly workaround!) my CI/CD pipeline by duplicating all related jobs and assigning them as single followers of the needed job with the OR condition.
The price tag for this workaround: the later jobs might be executed multiple times per pipeline (which might be unexpected, but acceptable behaviour sometimes, but sometimes it might be not acceptable, too?!).
Feature request
pre-build-job1:
script:
- echo "Prepare some stuff 1, but might fail"
build-job1:
needs: pre-build-job1
script:
- echo "Some stuff 1"
build-job2:
when: manual
script:
- echo "Some stuff 2"
post-build-job-with-or-condition:
needs: (build-job1 OR build-job2) # <= this line is NO VALID syntax and just demonstrates my requirement
script:
- echo "Do some stuff if job 1 or job 2 were successful
job-test:
needs: post-build-job-with-or-condition:
script:
- echo "Some testing"
job-deploy:
needs: job-test
script:
- echo "Deployment"
Very ugly workaround
pre-build-job1:
script:
- echo "Prepare some stuff 1, but might fail"
build-job1:
needs: pre-build-job1
script:
- echo "Some stuff 1"
build-job2:
when: manual
script:
- echo "Some stuff 2"
# Base implementation for jobs pipeline for build-job1 OR build-job2
post-build-job-with-or-condition-base:
script:
- echo "Do some stuff if job 1 or job 2 were successful
job-test-base:
script:
- echo "Some testing"
job-deploy-base:
script:
- echo "Deployment"
# Jobs pipeline for build-job1
post-build-job1-dependent:
needs: build-job1
extends: post-build-job-with-or-condition-base
job1-test:
needs: job1-dependent
extends: job-test-base
job1-deploy:
needs: job1-test
extends: job-deploy-base
# Jobs pipeline for build-job2
post-build-job2-dependent:
needs: build-job2
extends: post-build-job-with-or-condition-base
job2-test:
needs: job2-dependent
extends: job-test-base
job2-deploy:
needs: job2-test
extends: job-deploy-base
Same question here. We would like to have an “OR” condition for using “needs” or to have the possibility to set an “at least one” flag for the array of needs.
Same question here. In our case the use-case is a manual deploy job to one of three UAT environments. We would like to implement the “needs” relationship that deployment to one of the three UAT environments needs to have been successful for a production deployment to be allowed.
TL;DR; since it appears the conditional logic is not supported by needs, the solution I see for your case is to move Deploy_job to another pipeline that gets triggered by successful run of Test_job1 or Test_job2. The new pipeline would need to handle the case of both successful and “throttle” so that only the first one in is effective. See reference.
I came here from a similar but different need - to apply a condition to a GitLab CI Pipeline job’s needs - and so far I don’t see how it’s currently possible, since the documentation describes needs as a Job-level-only keyword which does not support any conditional parameter (like when).
In my case, I’ve got early build/push docker image stage/job that only runs when docker image dependencies change, i.e.:
docker-build-and-push:
stage: update-images
tags:
- shell
rules:
- changes:
- Dockerfile*
- .dockerignore
script:
- make docker-build docker-push
and a later stage which must be run afterwards, non-concurrently:
make-eks:
stage: create-management-cluster
tags:
- shell
# needs:
# job: docker-build-and-push
resource_group: pipeline_mutex
script:
- make eks
So I would like to uncomment that needs clause and of course this doesn’t work, syntactically nor functionally. The needs condition is needed because of the docker-build-and-push stage/job’s condition (the rules).
What I’ve arrived at that’s acceptable for now is to rely on GitLab CI stages’ normal sequential ordering, as described in Basic Pipelines. This works in my case because I have the luxury of being able to define the dependency as a single stage/job. Your situation seems a little different.
Sorry to not be offering a complete solution (yet), but maybe we can continue discussion and figure it out.
1 Like
there is a good solution available:
we can use optional for each job which we list under needs…
needs:
- job: docker-build-VITE
optional: true
- job: docker-build-WEBPACK
optional: true
a complete example:
docker-build-VITE:
extends: .docker
needs: ['docker-clean']
rules:
- if: $FRONTEND_TOOLING == "VITE"
script:
- export VITE_API_URL=$API_URL
- docker build --build-arg=VITE_API_URL --build-arg=NPM_PKG_AUTH_TOKEN --build-arg=FUNCTIONAL_USER_TOKEN --target production --tag $DOCKER_URL/$PROJ_GROUP_ID/$PROJ_ARTIFACT_ID:$PROJ_VERSION -f Dockerfile .
docker-build-WEBPACK:
extends: .docker
needs: ['docker-clean']
rules:
- if: $FRONTEND_TOOLING == "WEBPACK"
script:
- docker build --build-arg=NPM_PKG_AUTH_TOKEN --build-arg=FUNCTIONAL_USER_TOKEN --target production --tag $DOCKER_URL/$PROJ_GROUP_ID/$PROJ_ARTIFACT_ID:$PROJ_VERSION -f Dockerfile .
docker-login:
extends: .docker
needs:
- job: docker-build-VITE
optional: true
- job: docker-build-WEBPACK
optional: true
script:
- docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD $DOCKER_URL
2 Likes
I came here with the same question and this is exactly the solution I needed.
Thanks!
In my point of view, it would be sufficient if we could have an hybrid mode stage / dag :
Apart from the DAG feature that comes with needs, needs also provides the dotenv artifcat feature that is very convenient.
In addition this dotenv feature is often related to switching between develop / main branch.
The DAG would be built per stage and we could still rely on stage to define a preprocessing specific to develop / main branch.
Please find below a simplified example that is currently not working, but would be solved if I could use a per stage DAG instead of a per pipeline DAG:
variables:
CLUSTER_NAME: ""
TENANT_ENV: ""
stages:
- pre
- install
.create_dotenv:
stage: pre
script:
- echo ${CI_PIPELINE_SOURCE}
- echo "IKS_CLUSTER_NAME=${CLUSTER_NAME}" >> build.env
artifacts:
paths:
- build.env
reports:
dotenv: build.env
create_dotenv:
extends: .create_dotenv
only:
- tags
- main
create_dotenv_test:
extends: .create_dotenv
before_script:
- source .gitlab-ci/test.env
except:
- tags
- main
install:
stage: install
script:
- ./bootstrap.sh
# IMPOSSIBLE WITH GITLAB
# needs:
# - job: create_dotenv OR create_dotenv_test
# artifacts: true
and the testing variables .gitlab-ci/test.env are defined as :
# only export variable if it is undefined
export_if_void()
{
key=`echo $1 | awk -F '=' '{print $1}'`
if [[ -z ${!key} ]]; then
export $1
fi
}
export_if_void CLUSTER_NAME=my-test-cluster
export_if_void TENANT_ENV=test
This enables me to have a pipeline that fails on its own on main or tags because I want it to be only triggered by other pipelines.
However, in test branches it is ok to use a set of predefined variables pointing to test cluster.
I just saw your hint about the magic feature of needs : “optional: true” … thanks @tobiashochguertel
It solved my case above !
just needs:
install:
stage: install
script:
- ./bootstrap.sh
needs:
- job: create_dotenv
artifacts: true
optional: true
- job: create_dotenv_test
artifacts: true
optional: true
