Use artifacts from build job in deploy job, but only run if test stage passes

GitLab CI/CD
1

I’ve set up a pipeline with a deploy stage that uses artifacts from the build stage. Here is my (abbreviated) .gitlab-ci.yml:

stages:
  - build
  - test
  - docs
  - deploy

# Build stage
build/any:
  stage: build
  script:
    - # ...stuff...
  artifacts:
    paths:
      - dist

# Test stage
test/py38:
  stage: test
  needs:
    - build/any
  script:
    - # ...stuff...

# Deploy stage
deploy/pypi:
  stage: deploy
  needs:
    - build/any
  only:
    - /^dcc-\d(.\d){2}$/
  script:
    - # ...stuff...

I want to run the deploy job only if the intermediate test stage passes, but right now it’s running alongside the test stage jobs:

image
image751×268 24.7 KB

I think my use of needs: build/any in the deploy job is triggering it to run earlier than it should. I guess I could instead write:

# Deploy stage
deploy/pypi:
  stage: deploy
  needs:
    - build/any
    - test/py38
  only:
    - /^dcc-\d(.\d){2}$/
  script:
    - # ...stuff...

but in reality there are many test jobs and I’d prefer not to have to list them explicitly in needs in the deploy/pypi job config. Is there a way to have GitLab CI run the job only if all previous stages pass, but be given the artifacts from build/any? According to the documentation, stages should only run if all jobs in the previous stage pass, but it seems needs overrides this.

I am sure the solution is simple. Thanks in advance for any help!

2

I guess what I need here is a “needs: test” that requires a whole stage completes before triggering the job. I just found this open issue for GitLab that isn’t yet merged, so seems like this isn’t possible yet?

3

Yes, but you could add all of the jobs in the stage individually.

The other option is to remove needs altogether, and use rules instead of only, which would look something like this:

# Deploy stage
deploy/pypi:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH =~ /^dcc-\d(.\d){2}$/'
      when: on_success
    - when: never
  script:
    - # ...stuff...
4

Thanks. The rules approach seems even less clear than needs, so I think I’ll go for listing all required jobs in needs for clarity. Looking forward to being able to specify a stage in needs whenever (if ever) that feature lands…

5

So, just to be clear, rules is not an alternative to needs.

By default, every stage of the pipeline executes in the order you define them, i.e.

stages:
  - build
  - test
  - docs
  - deploy

Then in each job definition, rules or only/except can be used to stop jobs running under certain circumstances (i.e. only run the job if a file changes, or if the branch name matches a pattern).

needs does something slightly different, it allows a job to be executed earlier than the rest of the jobs in the stage, if the jobs listed in the needs matrix have already finished.

If you have jobs in the deploy stage, and you have the jobs from test in your needs matrix, then the deploy job can run before the docs stage has finished. But you could equally just move docs so that it runs after deploy if you want the jobs in that order.

6

Ok. It looks from the issue I linked above like it’s a long term goal of GitLab to make everything run using the directed acyclic graph (needs) approach, so I think I’ll keep the slightly hacky needs listing in anticipation that this will be well supported into the future.

1 Like