Best way to conditionally run multiple stages and jobs

GitLab CI/CD
1

What is the best way to manage stages and jobs that all run under similar conditions?

I have a pipeline definition where I would like to not run the entire test, build, deploy pipeline if the only changes are to the README, documentation, or other necessary parts of the repository.

I can see a possibility where I would want to have separate sub-pipelines, such as for document build/deploy jobs, but currently it’s just for the code.

I know of two possible implementations:

1. Implement only:changes for every test/build/deploy job.

I could fairly easily implement the following:

linting:
  stage: test
  only:
    changes:
      - .gitlab-ci.yml
      - "code/*.py"
      - ...
  script:
    [ commands ]

This works, and is clear, but has to be repeated on every single job, and this is going to be error-prone and will decrease readability. I have three stages in one project (test, build, deploy) and for each I have a development and a release job which are mostly identical. This means I have to repeat the above in six places.

2. Use child pipelines

I have developed and am currently using the following:

stages:
  - init

code:
  stage: init
  only:
    changes:
      - .ci-code.yml
      - "code/*.py"
      - ...
  trigger:
    include: .ci-code.yml
    strategy: depend

This works, and I like splitting out the jobs into different files (tidy!) so it’s good on the development side. On the GitLab UX side, though, I can’t see the whole pipeline in one view–it’s two pipelines, parent and child, so the child pipeline is hidden behind a box at first which can be expanded–but more seriously I now get two notifications when something fails or is fixed, because of strategy: depend. But I need that for the child’s status to be reflected in the parent’s.

I suspect the double-notifications are fixable and I just haven’t gotten there yet, but I’m still at the stage where I’m wondering if I’m heading down the right path.

Some other things I’ve thought of

  • An init job like the above that sets a variable which can then be checked by subsequent jobs.

    This doesn’t work because variables cannot be set and shared between jobs without the use of artifacts, and I fear that would make the pipeline look more complex.

  • Use of needs to say build won’t run unless test succeeds, and test job is gated with only:changes.

    That doesn’t work because GitLab determines that the build job could be set to run in cases where dependencies are not met–this is not how needs is meant to be used.

This can’t be an uncommon case, so is it a question of repository management, or is this just what we do? Thanks for any help.

2

A downside to the parent-child pipeline trigger that I haven’t overcome is that it must change artifact locations and my coverage badge as well as my todos badge (a custom badge) both stop functioning–the coverage badge reads unknown and the todos badge can’t be found.

Another downside I found is that only the parent pipeline’s stages are shown in the pipeline listing. In my definitions the parent only had a single stage which made the if/then decision on which way to go, so this didn’t impart as much information.

Having said that, the main issue I was trying to resolve was DRY–I didn’t want to keep repeating the list of files that could change in order for code-related CI jobs to run. I found a better way this morning: extends.

So now I’m back to (1) above but instead of repeating myself with the only: [files...] clause, I define a template job:

.code:
  only:
    changes:
      - .gitlab-ci.yml
      - "code/*.py"
      - ...

I then refer to this template in every job I want to if and only if the listed files are changed:

linting:
  extends: .code
  stage: test
  script:
    [ commands ]

This gives me the ability to stay DRY. To remain DRY. Ugh, I’m not trying to be clever there.

In addition, this is my preferred solution because:

  • the pipeline remains flat, so I can see all the jobs in the one pipeline window without extra clicks
  • similarly, every stage in the single pipeline are shown the pipeline listings
  • coverage and artifact-based badges work like I expect

I hope this helps somebody else. I’m still interested to know how others have dealt with these issues.

1 Like