Do not start new stage until manual run started by user at current stage

GitLab CI/CD
,
1

Hello,

I have this .gitlab-ci.yml on my gitlab.com repo:

cache:
  paths:
    - ~/.cache/

stages:
  - build
  - upload
  - download_test

build:
  image:
    name: python:3.9
  stage: build
  before_script:
    - echo aaa
    - echo "VERSION=123" >> version.env
  script:
    - echo bbb
  artifacts:
    paths:
      - "firmware.bin"
    reports:
      dotenv: version.env

.upload_job_template:
  image:
    name: amazoncorretto:21-al2-generic
  stage: upload
  needs:
    - job: build
      artifacts: true
  script:
    - cd ${CI_PROJECT_DIR}/.gitlab-ci/
    - echo ccc
  after_script:
    - echo ddd
  when: manual

upload_test:
  extends: .upload_job_template
  before_script:
    - echo test > "${CI_PROJECT_DIR}/.gitlab-ci/config.json"

upload_prod:
  extends: .upload_job_template
  before_script:
    - echo prod > "${CI_PROJECT_DIR}/.gitlab-ci/config.json"
  only:
    - master

.download_job_template:
  image:
    name: curlimages/curl
  stage: download_test
  parallel:
    matrix:
      - VARIANT:
        - XXX
        - YYY
  needs:
    - job: build
      artifacts: true
  script:
    - >-
      curl
      -siL
      -X GET
      --fail-with-body
      --show-error
      -H "x-abc: ${VARIANT}"
      "${URL}"
      -o firmware.bin
    - '[ $(stat -c%s "firmware.bin") -gt 512000 ] || exit 1'

download_test:
  extends: .download_job_template
  variables:
      URL: "http://dev.example.net/fw"

download_prod:
  extends: .download_job_template
  variables:
      URL: "http://prod.example.net/fw"

Výběr_528
Výběr_528681×272 16 KB

Výběr_529
Výběr_529601×350 21.7 KB

I want to run build after pipeline starts and after that manually run upload job. download_test must wait for upload job complete (after manual run). It’s not working. download_test job starts and upload_test still waiting for manual start.

How can I fix it?

Thanks.

2

The problem lies here:

.upload_job_template:
  needs:
      - job: build
        artifacts: true

From the documentation:

You can ignore stage ordering and run some jobs without waiting for others to complete.

I think you added needs because of the artifacts, right? You don’t have to! This is default behavior without needs.

By default, jobs in later stages automatically download all the artifacts created by jobs in earlier stages. You can control artifact download behavior in jobs with dependencies.

As a side note, I would advise not to use extends the way you use it. You make it much harder to read.

There are two options I can think of:

  • Use downstream Pipelines (with spec)
  • extend parallel:matrix for prod and dev

If you outsource 90% of the job configuration to a extends “template”, then you might rethink your strategy. I, personally, prefer a downstream pipeline because it’s much more readable.

3

Yes, I had needs because of artifacts.

I removed it:

cache:
  paths:
    - ~/.cache/

stages:
  - build
  - upload
  - download_test

build:
  image:
    name: python:3.9
  stage: build
  before_script:
    - echo aaa
    - echo "VERSION=123" >> version.env
  script:
    - echo bbb
  artifacts:
    paths:
      - "firmware.bin"
    reports:
      dotenv: version.env

.upload_job_template:
  image:
    name: amazoncorretto:21-al2-generic
  stage: upload
  script:
    - cd ${CI_PROJECT_DIR}/.gitlab-ci/
    - echo ccc
  after_script:
    - echo ddd
  when: manual

upload_test:
  extends: .upload_job_template
  before_script:
    - echo test > "${CI_PROJECT_DIR}/.gitlab-ci/config.json"

upload_prod:
  extends: .upload_job_template
  before_script:
    - echo prod > "${CI_PROJECT_DIR}/.gitlab-ci/config.json"
  only:
    - master

.download_job_template:
  image:
    name: curlimages/curl
  stage: download_test
  parallel:
    matrix:
      - VARIANT:
        - XXX
        - YYY
  needs:
    - job: build
      artifacts: true
  script:
    - >-
      curl
      -siL
      -X GET
      --fail-with-body
      --show-error
      -H "x-abc: ${VARIANT}"
      "${URL}"
      -o firmware.bin
    - '[ $(stat -c%s "firmware.bin") -gt 512000 ] || exit 1'

download_test:
  extends: .download_job_template
  variables:
      URL: "http://dev.example.net/fw"

download_prod:
  extends: .download_job_template
  variables:
      URL: "http://prod.example.net/fw"

but I still have the same problem:

Výběr_536
Výběr_536688×251 15.9 KB

What else could be wrong? I also tried ChatGPT but also without success.

4

I fixed it!

Just add allow_failure: false to .upload_job_template.

There is complete working pipeline:

cache:
  paths:
    - ~/.cache/

stages:
  - build
  - upload
  - download_test

build:
  image:
    name: python:3.9
  stage: build
  before_script:
    - echo aaa
    - echo "VERSION=123" >> version.env
  script:
    - echo bbb
  artifacts:
    paths:
      - "firmware.bin"
    reports:
      dotenv: version.env

.upload_job_template:
  image:
    name: amazoncorretto:21-al2-generic
  stage: upload
  script:
    - cd ${CI_PROJECT_DIR}/.gitlab-ci/
    - echo ccc
  after_script:
    - echo ddd
  when: manual
  allow_failure: false
  
upload_test:
  extends: .upload_job_template
  before_script:
    - echo test > "${CI_PROJECT_DIR}/.gitlab-ci/config.json"

upload_prod:
  extends: .upload_job_template
  before_script:
    - echo prod > "${CI_PROJECT_DIR}/.gitlab-ci/config.json"
  only:
    - master

.download_job_template:
  image:
    name: curlimages/curl
  stage: download_test
  parallel:
    matrix:
      - VARIANT:
        - XXX
        - YYY
  needs:
    - job: build
      artifacts: true
  script:
    - >-
      curl
      -siL
      -X GET
      --fail-with-body
      --show-error
      -H "x-abc: ${VARIANT}"
      "${URL}"
      -o firmware.bin
    - '[ $(stat -c%s "firmware.bin") -gt 512000 ] || exit 1'

download_test:
  extends: .download_job_template
  variables:
      URL: "http://dev.example.net/fw"

download_prod:
  extends: .download_job_template
  variables:
      URL: "http://prod.example.net/fw"
1 Like