I’m attempting to run jobs in a matrix (parallel:matrix) using a build time variable.
In the pipeline, I attempt to read a descriptor, e.g., POM, with yq and with the values collected set a variable.
This variable is provided to the parallel:matrix (where $MAVEN_JIB_MODULES is the result of $(yq -ocsv '.project.modules.module' $CI_PROJECT_DIR/pom.xml) which has been validated and works), i.e.,
parallel:
matrix:
- IMAGES: [ "$MAVEN_JIB_MODULES" ]
I have genuine suspicion that the issue is in the pipeline lifecycle. On pipeline creation, the parallel:matrix is provided an empty variable. The only way to effectively do this is defining the variable, e.g., MAVEN_JIB_MODULES, in the .gitlab-ci.yml, statically.
Correct me if I’m wrong.
Working fork… TBC Maven Jib Matrix Variant
I found a couple of reports in the issues of folks with similar problem. The issues were linked off of this post in the forums. I was able to get the form $[[ inputs.jib-maven-modules ]] to expand into the flow sequence, i.e., - IMAGES: [ value, value ] but this seem to resolve as a string and produced only one job.
If I find something that works, I’ll followup. Right nothing seems “natively” supported and most are using workarounds.
I found a lot of help in the documentation , of all places! Go figure.
My first issue was I was attempting to write a YAML flow sequence to a pipeline variable, e.g., variables:FOO. This was returning an error that I was incorrect attributing to the parallel:matrix definition. I moved this definition into a shell script and exported it to the environment for downstream jobs (via dot env files). This is accomplished by parsing the component array, e.g., $(echo $[[ inputs.jib-maven-modules ]] | yq -ocsv '.' -) and echoing that out to the dot environment file artifact.
Since I can’t dynamically, determine the Maven modules I want to build from the pom.xml, i.e., $(yq -ocsv '.project.modules.module' $CI_PROJECT_DIR/pom.xml) (I mean that dynamically determines them, but as noted previously, the pipeline lifecycle is such that I can’t see this on pipeline creation… it least I haven’t found a way).
That said, with the environment, my downstream jobs can now get the modules to build and I can build the matrix for registry deployment and release.
The repository is still a work in progress, lots of cruft and brutishness, but I clean it up and it will go through a thorough review before released to the upstream repository of CI/CD components in the catalog.