Why do we need merge request pipelines?

Until today* I thought I could get away with not having dedicated merge request pipelines at all and only rely on regular branch pipelines. That way I could avoid the dreaded “multiple pipelines” and “detached pipelines” issues.

Usually things in GitLab are there for a good reason. So, I kept and still keep wondering: why would I need merge request pipelines? Why were they introduced in the first place? At what point in the development lifecycle should I consider using them?

We follow the “stable master” pattern where every push to master is auto-released. What I can imagine in terms of lifecycle wrt MR pipelines:

  1. start a new branch and push commits: have a (minimal) pipeline, maybe just “compile” (where applicable) and test
  2. when done offer the branch as a MR: have a more elaborate merge request pipeline that might include things like secret detection, static code analysis, further consistency checks in addition to the jobs in 1. thereby trying to avoid having two pipelines for every commit
  3. once MR is merged: run the release pipeline

Currently we only have (1+2) & 3 thereby not distinguishing between branch and MR builds.

backend:continuous:
  ...
  only:   # on all branches but not on tags
    - branches
  except: # not on master, release and RC branches
    refs:
      - master
      - /^release-\d+\.\d+\.x$/
      - /^rc-\d+\.\d+\.\d+$/
...
backend:
  ...
  only:   # for release builds i.e. not on development branches
    - master
    - /^release-\d+\.\d+\.x$/
    - /^rc-\d+\.\d+\.\d+$/

* the reason why this changed is that SonarQube seems to require merge request pipelines in order to offer its MR decoration feature

Hi, @marcelstoer.

why would I need merge request pipelines? At what point in the development lifecycle should I consider using them?

From my own perspective, MR pipelines can provide significant cost savings for large/busy projects. With basic pipelines, every commit triggers a new pipeline run.

Why were they introduced in the first place?

I don’t know, but here’s some historical context that I found:

  1. `only/except: merge_requests` for merge request pipelines (#15310) · Issues · GitLab.org / GitLab FOSS · GitLab
  2. Better management of stability in master (&101) · Epics · GitLab.org · GitLab

While I work for GitLab, this is not one my areas of expertise. Hopefully someone else chimes in with more insights.

Minor correction…

Not every commit but every push to the respective branch i.e. you push 10 commits and GitLab creates a pipeline based on the last commit. AFAIU this behavior is no different with MR pipelines.

@marcelstoer

Not every commit but every push to the respective branch

Yes! Thanks for clarifying. The scenario I had in mind was multiple people pushing to multiple branches, like we have in GitLab.

AFAIU this behavior is no different with MR pipelines.

Ah, good point: it depends on the workflow being used. If you choose to create a draft MR along with a new branch, they do seem the same. :thinking:

Using MR pipelines allows job to use predefined variables for MR pipelines in CI - the ones starting with CI_MERGE_REQUEST_*: Predefined variables reference | GitLab . I imagine that’s why sonarqube requires it. Without those, there’s no way to identify what’s the target branch or MR id. Without MR id there’s no way to automatically post comment to MR.

There are also some merge features that require merge request pipelines, like pipelines for merged result (Pipelines for merged results | GitLab) or merge trains.

I’m not sure why GitLab did it this way and whether it was neccessary to split it that way. Once I got used to it and the weird detached name, I don’t mind it that much. When we were transitioning to MR pipelines, I’ve simply set up workflow, and I didn’t have to change anything else in my jobs. With proper workflow specified, you won’t get duplicate jobs.

In general, I think that setting up a good workflow.rules (GitLab CI/CD `workflow` keyword | GitLab) is the key to getting used to that feature and getting most out of it.

Thanks for your feedback!

Actually it doesn’t (or so I claim) as it allows to set the parameters sonar.pullrequest.key, sonar.pullrequest.branch, sonar.pullrequest.base for MR decoration manually. I did but it still refuses to decorate my MRs unless the scanner is triggered from an MR pipeline. But that’s something for their forum rather :slightly_frowning_face:

Thanks, after all this digging I now arrived at the same conclusion.