Why "needs" is not transitive (at least for artifacts)

I have init job on the begging of my pipeline which product dotenv artifact. I want to use generated variables in all other jobs. This solution was fine unless I started to use “needs” for job dependencies. I realised, that when I have tree jobs:

  • init (produces dotenv artifact)
  • build (needs init)
  • deploy (needs build)

the deploy job don’t see the dotenv artifact until I explictily set needs to init.

As this is only example, in reality, the pipeline consist of much more jobs with various dependencies.

Should I put the dependency on init to all the jobs? Why gitlab-ci doesn’t count with “transitivity”?

1 Like

I spent a whole lot of time banging my head against the wall, trying to understand this exact issue you’re describing.

The pipeline I’m building has a similar structure to yours, with a job in the init stage that authenticates against an AWS ECR to then store the password in a file managed as an artifact so subsequent stages can use it to push and pull images, without having each of the jobs authenticate each time – thus making the pipeline more efficient.

I can think of several ways to work around this, and the one I like best so far involves using an S3 bucket to store our artifacts.

Another solution would be implementing a block storage to avoid wasting time interacting with the bucket, but this approach will require further research… :roll_eyes:

Anyway, I created an issue and cited your example, @martin.bery.beranek: