Matrix Builds in CI

GitLab CI/CD
#1

I typically use a personally hosted version of https://github.com/drone/drone for my CI (with Travis and Appveyor for OSX/Windows builds) but I thought I’d give gitlab CI a run.

However, I’ve found defining a matrix build to be hugely complicated. Is there a simpler way? The docs discuss using anchors for templates.

Contrast with drone / travis / appveyor which let me define a section like this

matrix:
  SCALA_VERSION:
    - 2.12.3
    - 2.11.11

which means that the entire pipeline runs again with the environment variable set as defined here.

Yes this is not very powerful for more advanced builds, but it is a critical part of cross building in scala (which has multiple active versions).

2 Likes
#2

I find it a bit clunky but matrix jobs are generally accomplished via YAML anchors: https://docs.gitlab.com/ee/ci/yaml/#anchors

#3

While, technically, using YAML anchors is a solution there is a lot of boilerplate code with this. Which compromises readability.

We have pipelines with 3x4 components to mix, which requires us to write 12 chucks of the same code with just a label and a value different in each. Apart from the code not being particularly elegant and violating the DRY principle, maintaining it is severely error prone. To avoid human error we try to generate this code where it makes sense. All in all, this is a huge pain. Matrices should look like the example quoted by @fommil above to provide an abstract view on a concrete problem hiding the implementation details.

Complicated things must be easy to read! To allow us to get more complicated things done. With ease.

Related Issues

#4

You, of course, completely correct. I agree with you. Hopefully the extends keywords with its deep-merge super-powers will ease things up. Coming with 11.3 on the 22nd or on an RC release in a repo next to you. :slight_smile:

https://docs.gitlab.com/ee/ci/yaml/README.html#extends

1 Like
#5

Good news and bad news:

  • The extends keyword is there, and it makes extending jobs nice, without YAML anchors.
  • Matrix builds will not be implemented using a Travis-alike syntax but by via child/parent pipelines.
#6

Good news! Matrix builds are now possible using the parallel: matrix: keyword hierarchy.

The visual implementation is different though compared to doing it “manually”, which some people may like and some may find cumbersome.

Example (traditional)

image
image897×343 21.9 KB

We can use extends to reduce excessive repetition a bit, but in essence it’s all copy+paste:

.check:
  stage: check
  image: docker.io/painless/tox

flake8:
  extends: .check
  script: tox -e flake8

pylint:
  extends: .check
  script: tox -e pylint

...

Example (matrix)

image
image903×112 7.95 KB

image
image896×303 20.4 KB

With the matrix keyword we can do it all in a single block, but also the result is visually grouped. (The screenshot above below is when you click on the job group box.)

checks:
  stage: checks
  image: docker.io/painless/tox
  script: tox
  parallel:
    matrix:
    - TOXENV:
      - flake8
      - pylint
      - bandit
      - safety
      - kubernetes

Traditional layout possible?

Does anyone know whether it’s possible to get the traditional layout also with the new matrix syntax?

1 Like