CICD pipeline - run the dev stage manually on all branches except main

I have 4 stages in my gilab cicd pipeline -

stages:

  • dev-deploy
  • test-deploy
  • approval
  • prod-deploy

there are 4 jobs inside the stage dev-deploy

I want to run the dev-deploy stage alone for all branch except main (which also the default branch).

The rule I have for all 4 jobs inside the stage dev-deploy is :

  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never

So on every push to any branch except main - I should be able to create a pipeline which shows only the dev-deploy stage and not the other stages and it should be MANUAL.

The above rule works fine only for the first time I push a change from my local branch to the remote branch.

for example, I create a branch on my local “x/foo” commit some changes and push to the remote for the first time then the pipeline is created and I can see the dev-deploy stage in the new pipeline created. But if I make additional commits to the same branch and push again I do not see a new pipeline with the dev-deploy stage in it.

I do not want to run the dev-deploy on any sort of MRs.

complete flow :

dev-deploy-common:
  stage: dev-deploy
  script:
    - cd ./aws_code_base/platform/CommonLib/
    - python setup.py setopt --command=options.package_data --option=CommonLib --set-value=devConfigs.json
    - python setup.py sdist
  artifacts:
    name: dev-commonlib
    paths:
      - aws_code_base/platform/CommonLib/dist/*tar.gz
    expire_in: 1 day
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - aws_code_base/platform/emr_on_eks/Dockerfile
        - aws_code_base/platform/emr_on_eks/Notebook.Dockerfile
        - aws_code_base/platform/CommonLib/**
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  tags:
    - dev
    - aws

dev-deploy-emrcustomimage:
  stage: dev-deploy
  script:
    - cd ./aws_code_base/platform/
    - aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 123********.dkr.ecr.us-west-2.amazonaws.com
    - docker pull 123********.dkr.ecr.us-west-2.amazonaws.com/spark/emr-6.11.0:latest
    - docker pull 123********.dkr.ecr.us-west-2.amazonaws.com/notebook-jeg/emr-6.11.0:latest
    - docker pull 123********.dkr.ecr.us-west-2.amazonaws.com/notebook-spark/emr-6.11.0:latest
    - cp $CI_PROJECT_DIR/aws_code_base/platform/CommonLib/dist/*tar.gz .
    - docker build --no-cache -f ./emr_on_eks/Dockerfile -t emr6.11.0_custom .
    - docker build --no-cache -f ./emr_on_eks/Notebook.Dockerfile -t emr6.11.0_notebook_custom .
    - aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin {account_id}.dkr.ecr.us-west-2.amazonaws.com
    - docker tag emr6.11.0_custom {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:dev
    - docker tag emr6.11.0_notebook_custom {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:notebook_dev
    - docker push {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:dev
    - docker push {account_id}.dkr.ecr.us-west-2.amazonaws.com/emr6.11.0_custom_repo:notebook_dev
    - IMAGES_TO_DELETE=$( aws ecr list-images --region us-west-2 --repository-name emr6.11.0_custom_repo --filter "tagStatus=UNTAGGED" --query 'imageIds[*]' --output json )
    - aws ecr batch-delete-image --region us-west-2 --repository-name emr6.11.0_custom_repo --image-ids "$IMAGES_TO_DELETE" || true
    - docker system prune -a -f
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - aws_code_base/platform/emr_on_eks/Dockerfile
        - aws_code_base/platform/emr_on_eks/Notebook.Dockerfile
        - aws_code_base/platform/CommonLib/**
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  needs:
    - job: deploy-dev-commonlib
      artifacts: true
  tags:
    - dev
    - aws

dev-deploy-postgres:
  stage: dev-deploy
  script:
    - pip3 install boto3
    - git fetch
    - added_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - modified_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - renamed_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - echo $added_files "," $modified_files "," $renamed_files
    - python3 aws_code_base/platform/deployments/scripts/postgres_deploy.py "$added_files" "$modified_files" "$renamed_files" "aws_code_base/platform/CommonLib/CommonLib/devConfigs.json"
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - 'aws_code_base/platform/postgres/**'
        - 'aws_code_base/dataengineering/**/schema_*.json'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  tags:
    - dev
    - aws

dev-deploy-snowflake:
  stage: dev-deploy
  script:
    - pip3 install boto3 snowflake-connector-python --use-feature=2020-resolver
    - git fetch
    - added_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - modified_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - renamed_files=$(git diff-tree --no-commit-id --name-only -r $CI_COMMIT_SHA | tr '\n' ' ')
    - echo $added_files "," $modified_files "," $renamed_files
    - python3 aws_code_base/platform/deployments/scripts/snowflake_deploy.py "$added_files" "$modified_files" "$renamed_files" "aws_code_base/platform/CommonLib/CommonLib/devConfigs.json" "dev"
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: false
      changes:
        - aws_code_base/snowflake/**
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: never
  tags:
    - dev
    - aws

Two main things here:

  1. I should be able to build the pipeline on every commit to the branch with only one stage (dev-deploy) and should be able to run manually if I want to. this should not build a pipeline when there is open MR.

  2. Even if I run manually - the jobs inside the stage dev-deploy should adhere to it’s changes section. Which means it should only run when it encounters a change in those directories/files.