How to properly implement a multi-project pipeline with 2 upstreams projects

How to properly implement a multi-project pipeline with 2 upstreams projects

Bit of context:

The delivered application is made of multiple components, each one versioned separately and then having its own pipeline building the artifacts.

These artifacts must be packaged together to create the deliverable.

From my understanding, making use of a multi-projects pipeline could do the trick, by making use of
needs:projects

On the downstream pipeline, I then would like to have:

downstream-pipeline-ab:
  variables:
  needs:
    - project: upstream-project-a
      job: upstream-job-a
      ref: $UPSTREAM_REF_A # declared as variable in the trigger-job-a
      artifacts: true
    - project: upstream-project-b
      job: upstream-job-b
      ref: $UPSTREAM_REF_B # declared as variable in the trigger-job-b
      artifacts: true

However, it seems impossible to both provision UPSTREAM_REF_A and UPSTREAM_REF_B as the downstream pipeline is obviously launched by one upstream at the time not both.

What I’m doing wrong ? How should I do to get the latest artifacts from pipeline A and B ?

Does simply multi-projects allow to fetch artifacts from multiple repos ?
And if not, what would be the gitlab-way to implement something solving my initial problem ?

Thank you in advance !

@simba-oh Welcome to the forum and thanks for the question!

I’m not clear what the problem is you are running into from your post. It sounds like when one upstream project builds it kicks off the downstream project and that is NOT the behavior you want maybe? if so can you confirm or clarify what behavior you are looking for to help me understand?

Thanks!

-James H, GitLab Product Manager, Verify:Pipeline Execution

It sounds like when one upstream project builds it kicks off the downstream project and that is NOT the behavior you want maybe

I expect the downstream pipeline to be kicked off, and fortunately it is.

Let me rephrase what I want to achieve first:

1) I would like either pipeline A or pipeline B to trigger pipeline C (downstream).
2) I would like to use the artifacts generated in A or B in C
3) C needs artifacts from both A and B at the same time (as it packages the whole application itself).

It is easy to achieve 1) and 2)

I simply wrote a trigger task in A:

job-a:
  stage: build
  # Build my artifacts
  # ...

trigger-c-from-a:
  stage: deploy
  variables:
    UPSTREAM_REF_A: $CI_COMMIT_REF_NAME # make the ref available in C
  needs: [ 'job-a' ]
  trigger:
    project: path-to-project-c
    strategy: depend

And in my project B, I have a similar snippet.

And for C:

downstream-job-c:
  variables:
  needs:
    - project: upstream-project-a
      job: job-a
      ref: $UPSTREAM_REF_A
      artifacts: true
    - project: upstream-project-b
      job: upstream-job-b
      ref: $UPSTREAM_REF_B
      artifacts: true

As you can see, UPSTREAM_REF_A will be passed to the downstream pipeline according to the doc. This exact variable defined upstream and used downstream for the commit reference mechanism is even described in the same section.

Achieving my 3) seems impossible using that mechanism.

When trigger-c-from-a is executed, only UPSTREAM_REF_A will be filled, then my downstream pipeline will fail due to expired/erased artifacts (UPSTREAM_REF_B being an invalid reference).

Of course, same reasoning applies for trigger-c-from-b where UPSTREAM_REF_A will be invalid, leading to the same error.

So, how should I proceed to make my downstream-pipeline aware of all the references I need when triggered if passing them by variable is not working ?

Creating a dotenv file (variable inheritance) shouldn’t work too, so it seems it is’nt a good approach.

Maybe I’m failing for the XY problem and there is another approach to package artifacts from multiple repositories in a gitlab-way which does not make use of the needs keyword ?

I’m sorry for the wall of text but I hope it better describes what I’m facing and what I would like to achieve.

@simba-oh - Thanks so much for the extra detail!

I don’t think you can achieve what you want directly with needs:project, this link has some info about that.

What you might be able to do is use the API to get the file. I found a writeup about this here.

Hope this helps!

-James H, GitLab Product Manager, Verify:Pipeline Execution

@jheimbuck_gl - Sorry for the late answer, was busy for the rest of the week !
Thank you for the writeup, I’ll have a look and see if it solves my problem.