Worflow rules for both MRs an conventional branches

Hello,

The project I work on has moved to Gitlab only a couple of weeks ago, and I could use some help with applying workflow rules in the pipeline definition.

With the change to Gitlab CI/CD, we have adopted the Gitflow workflow. (We might develop further in the future, but this we thought was a good starting point.) So there is a branch we call ‘dev’ where all development is done and there is a ‘master’ branch for the official releases of the project. Merge requests are created with respect to the ‘dev’ branch. When a release is approaching, we branch off from ‘dev’, finish the release and merge the branch both into ‘dev’ and into ‘master’. Hence, we need pipelines for both Merge Requests and for conventional branches.

Generally, pipelines shall trigger only when changes are pushed into the repository. Especially, MR creation and branch creation shall not trigger any pipelines in order to save build time. The reason for this is our presently limited capacity of the build resources.

In order to establish the desired behaviour, I have (naively) tried a workflow definition as follows:

workflow:
  rules:
    - if: $CI_MERGE_REQUEST_IID
      changes:
      - "**/*"
    - if: $CI_COMMIT_BRANCH 
      changes:
      - "**/*"

However, it doesn’t work as desired. When a Merge Requests without any changes is created, a pipeline gets triggered. When a conventional branch without any changes is created, a pipeline is triggered. I have tried many different twists and variations, but haven’t been able to find a definition satisfying the needs described above.

Even though I have studied the official documentation describing pipelines, workflows and rules carefully, reading it all over again and again, it seems that I am still missing something somewhere.

What would be the “right” way to setup workflow rules that work for both MRs and conventional branches?

Any help would be highly appreciated.

Thank you for your help
Marko

p.s. Gitlab version used is 12.8.6

Hello,
Any help would still be very appreciated.

Hi @ChildsEye! Welcome to our forum! I first want to apologize: you have been waiting since June 8 for a response from us. I must have missed this altogether - sorry about that!

Ok, I am going to take this straight to experts on setting up workflow rules. Can you link me to the docs you have already checked out so we don’t end up re-sending them over? Thanks!

Hello @Linds,
I am sorry for the delayed response, I was on an extended leave from my workplace until yesterday.

As for the workflow rules, I have studied official documentation quite intensively (also following all the links), this being the main source of wisdom.

Additionally I have also tried searching in this forum (this post looked promissing back then) as well as googling the internet, which would typically point to the same pages all over again.

Thank you for your effort in advance!

Best regards,
Marko

Hi @ChildsEye

in my repos, I also only want to trigger one pipeline for a feature branch, so I have something like this at the top of all my CI configs:

workflow:
    rules:
        - if: $CI_MERGE_REQUEST_ID
          when: never
        - if: $CI_COMMIT_TAG != null
          when: never
        - when: always

So, we don’t trigger a second pipeline for MRs and we don’t automatically trigger a pipeline when tags are pushed, but anything pushed to a feature branch (or merged into dev or master) gets a pipeline.

Personally, I would tend to stay away from changes, but when I build Dockerfiles I generally have something like this:

docker:
    stage: prepare
    ...
    rules:
        - if: '$CI_PIPELINE_SOURCE == "schedule"'
          when: never
        - changes:
              - .meta/Dockerfile
          when: always
        - when: never 

just because Docker builds are quite expensive in terms of CI time and I don’t want to wait around for them on every single push to a feature branch (but I do want to build the Dockerfile once at the start of every MR, at the very least).

The '$CI_PIPELINE_SOURCE == "schedule"' test means that scheduled pipelines run on the latest container that was built in the default branch.

Does that help?

Sarah

Hello @snim2,

Thank you for the suggestion, I will try it out and report the results here.

Best regards,
Marko

Hi @snim2,
Finally, I got a chance to try out your suggestion. We have done some repository restructuring, which now allows us to experiment freely.

The workflow rules suggestion works fine for me.

I still have one related question. Is it possible to specify workflow rules such that pipeline creation is based on files changed? For instance, when a documentation file is updated and pushed (README.md being a canonical example), I don’t want a pipeline to get instantiated. Of course, when source code is changed, I want a pipeline to run.

Is there a way to implement such a behaviour?

Both your comment and the official documentation suggest that changes clause pertains to individual jobs rather than workflow rules. Is that correct?

Best regards,
Marko

Hi @ChildsEye,

glad this is working for you! Yes, you are essentially right, but just to be clear, a pipeline is any jobs that run in your CI, so what you want to do isn’t really to “not always have pipelines” it’s more to change the pipeline jobs you have depending on the files that are changed.

A pipeline can have many stages (which run one after the other) and each stage can have one or more jobs that may run either in parallel or sequentially.

I think for your scenario you might want something like this (I haven’t run this through a lint, so there may be mistakes):

workflow:
    rules:
        - if: $CI_MERGE_REQUEST_ID
          when: never
        - if: $CI_COMMIT_TAG != null
          when: never
        - when: always

stages:
    - lint
    - build
    - test
    - ...

lint:markdown:
    stage: lint
    rules:
        - changes:
              - README.md
              - CONTRIBUTING.md
          when: always
        - when: never 
    script:
        - mdl *.md

lint:java:checkstyle:
    rules:
        - changes:
              - README.md
              - CONTRIBUTING.md
          when: never
        - when: always 
    script:
        - mvn clean checkstyle:check

build:java:
    stage: build
    rules:
        - changes:
              - README.md
              - CONTRIBUTING.md
          when: never
        - when: always 
    script:
        - mvn build
    artifacts:
        paths:
            - target/

test:java:spotbugs:
    stage: test
    rules:
        - changes:
              - README.md
              - CONTRIBUTING.md
          when: never
        - when: on_success
    dependencies:
        - build:java
    script:
        - mvn spotbugs:check

So, here we have three stages: lint, build, test. The lint stage has two jobs, one which lints markdown files and one which lints Java files.

There are two quick things to note here. One is that this is a bit long winded! If you wanted to tidy it up, a good place to start would be to add some anchors to remove some of the duplication.

Secondly, depending on how your code base is structured, this might seem a bit long-winded, for example if you end up with a lot of nested directories to list in the rules:changes clauses. You can see some more examples in this issue.

Thirdly, I’ve noted here that some jobs depend on others, i.e. the test:java:spotbugs job depends on build:java. By doing this and using on_success you can ensure that if the build stage fails, the test stage will not be run. Listing artifacts here ensures that the results of the build stage are passed to the test stage, so test does not have to build again.

If you write a nice workflow like this, but you still find your pipeline is running slow, the last thing to look at would be caching.

Hope that helps a little!

Sarah

Hello @snim2,

One last comment on this issue.

In one of our repositories, we very much needed the option to not run the whole build pipeline (which takes some time) depending on some files being changed. In order to sove this we employed the parent-child pipeline feature. So the general pipeline consists of a signle stage with two mutually exclusive jobs. One job triggers the build pipeline when appropriate, the other job runs when the build pipeline is not needed, only doing some output, and ends quickly.

Thank you for your time and help.

Best regards,
Marko