First manual stage being skipped

Problem to solve

I am wanting a pipeline that allows one job in a stage of multiple jobs, to be manually run to set variables downstream.

I am converting a project from being a Zappa deployed project, to a Terraform deployed project.
This project can deploy to multiple environments with Zappa. In order to reproduce that ability in CICD using terraform, I need to designate the environment early in the pipeline, since downstream jobs will depend upon the environment designation.

My idea was to have a beginning stage, with one job for each environment. These would be manual jobs, each job would set an environment variable dictating which environment to deploy to. The next stage would depend upon at least one of those jobs completing successfully.

What is happening, is the manual steps are skipped and the next job runs automatically.

Steps to reproduce

Create a simple pipeline with two stages, similar to this example

stages:
  - build
  - test

# Stage 1: Build
# These jobs need to be triggered manually
build_job1:
  stage: build
  script:
    - echo "Running build_job1"
    - exit 0  # Simulate success
  when: manual  # Requires manual execution

build_job2:
  stage: build
  script:
    - echo "Running build_job2"
    - exit 0  # Simulate success
  when: manual  # Requires manual execution

build_job3:
  stage: build
  script:
    - echo "Running build_job3"
    - exit 1  # Simulate failure
  when: manual  # Requires manual execution

# Stage 2: Test
# This job will run only if build_job1 or build_job2 succeeds
test_job:
  stage: test
  script:
    - echo "Running test_job"
  needs:
    - job: build_job1
      optional: true
    - job: build_job2
      optional: true
    - job: build_job3
      optional: true
  when: on_success

Running this pipeline will cause both stages to be skipped.

Configuration

Here is a truncated version of the pipeline I’m working in

stages:
  - environment
  - linting
  - unittest
  - prepare
  - validate
  - bundle
  - plan
  - deploy

deploy_dev:
  stage: environment
  variables:
    TARGET_ENV: "dev"
    ENVIRONMENT_VARIABLE: "nonprod"
  script:
    - echo ${TARGET_ENV}
  when:
    manual
  tags:
    - aws
    - nonprod


deploy_nonprod:
  stage: environment
  variables:
    TARGET_ENV: "nonprod"
    ENVIRONMENT_VARIABLE: "nonprod"
  script:
    - echo ${TARGET_ENV}
  when:
    manual
  tags:
    - aws
    - nonprod

deploy_staging:
  stage: environment
  variables:
    TARGET_ENV: "staging"
    ENVIRONMENT_VARIABLE: "prod"
  script:
    - echo ${TARGET_ENV}
  when:
    manual
  tags:
    - aws
    - nonprod

deploy_prod:
  stage: environment
  variables:
    TARGET_ENV: "staging"
    ENVIRONMENT_VARIABLE: "prod"
  script:
    - echo ${TARGET_ENV}
  when:
    manual
  tags:
    - aws
    - nonprod

variables:
  TF_ROOT: ${CI_PROJECT_DIR}/terraform
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${ENVIRONMENT_VARIABLE}
  AWS_REGION: us-east-1

.base_terraform:
  image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
  before_script:
    - cd ${TF_ROOT}
    - gitlab-terraform --version
    - echo "Selecting backend file for ${ENVIRONMENT_VARIABLE}"
    - mv backend.tf.${ENVIRONMENT_VARIABLE} backend.tf
  cache:
    key: ${CI_PROJECT_NAME}
    paths:
      - ./terraform/.terraform
  tags:
    - aws
    - ${ENVIRONMENT_VARIABLE}

.base_python:
  image: "python:3.12"
  tags:
    - aws
    - ${ENVIRONMENT_VARIABLE}

black:
  extends: .base_python
  stage: linting
  script:
    - pip install black
    - black --check .
  needs:
    - job: deploy_dev
      optional: true
    - job: deploy_nonprod
      optional: true
  when: on_success  # Ensures it only runs if the jobs it needs have succeeded

flake8:
  extends: .base_python
  stage: linting
  script:
    - pip install flake8
    - flake8 --statistics --count --max-line-length=88 .
  needs:
    - job: deploy_dev
      optional: true
    - job: deploy_nonprod
      optional: true
  when: on_success  # Ensures it only runs if the jobs it needs have succeeded

Versions

Please select whether options apply, and add the version information.

• Self-managed
• GitLab.com SaaS
• Dedicated
• Self-hosted Runners

Versions

• GitLab : Unknown, don’t have access to find out
• GitLab Runner: 15.11.0