Running on GitLab Enterprise Edition [15.3.3-ee] company self hosted.
I have a number of large projects that have linear (non-circular) dependancies; libraries, programs, installs, … all for desktop applications.
Each will build in their pipeline correctly when all are up to date, but if a push is made to a program project that relies on a change in a library project that wasn’t push, it will fail.
In order to solve this we have policy to push projects in a specific order which generally works.
We want a complete full build of everything done overnight every night and I’ve gone the route of a scheduled pipeline that triggers all the others in order (parallel when can) but immediately ran into build failures which seems inexplicable when I could run a pipeline on the exact same thing and it worked. I finally figured out that the log when it failed showed that the git clone was being skipped.
I tracked it down to the fact that we use a lot of template yaml files that are included to define functions for script in the before_script to make the scripts much shorter and focused. And because the project repos are generally very large, any pipeline job that doesn’t need the repo (which is pretty much all but the build stage/job) has a GIT_STRATEGY of ‘none’ set in their template. The pipeline that is triggering the others doesn’t need anything in the jobs that are triggering the others, yet that setting propagates to the triggered pipeline overriding its setting whether inherited or specifically set.
I created 2 sandbox projects just to test this and show the “problem”.
Here is my test sandbox yaml:
include:
- project: 'great-bay/automation'
ref: main
file: '/YamlTemplates/.global_template.yml'
default:
before_script:
- !reference [.global_template, before_script]
stages:
- build
.build_template:
variables:
GIT_STRATEGY: clone
build1:
stage: build
variables:
GIT_STRATEGY: clone
script: |
if( $Env:GIT_STRATEGY -eq 'clone' )
{ LogMessageGreen "GIT_STRATEGY is 'clone' as expected" }
else
{
LogMessageRed "GIT_STRATEGY is '$Env:GIT_STRATEGY', not 'clone' as expected"
Exit 1
}
build2:
extends: .build_template
stage: build
script: |
if( $Env:GIT_STRATEGY -eq 'clone' )
{ LogMessageGreen "GIT_STRATEGY is 'clone' as expected" }
else
{
LogMessageRed "GIT_STRATEGY is '$Env:GIT_STRATEGY', not 'clone' as expected"
Exit 1
}
Here is the triggering sandbox2 yaml:
stages:
- trig
.trig_template:
variables:
GIT_STRATEGY: none
trig1:
stage: trig
variables:
GIT_STRATEGY: none
trigger:
project: great-bay/Sandbox
branch: test_trigger_vars
strategy: depend
trig2:
extends: .trig_template
stage: trig
trigger:
project: great-bay/Sandbox
branch: test_trigger_vars
strategy: depend
trig3:
stage: trig
trigger:
project: great-bay/Sandbox
branch: test_trigger_vars
strategy: depend
Only ‘trig3’ succeeds.
Note, this is pared way down as normally a build stage would extend the .build_template which extends the .global_template, etc. And those templates have the GIT_STRATEGY set.
The only way for me to solve it is to NOT set a git strategy in any way in the parent pipeline, but does it waste time getting the repo? I can’t tell since I can’t see a log for those jobs and the documentation doesn’t really say what is happening.
Doing so will mean I can’t import any of my templates, but to do that I may need an entirely separate project so it has a clean/empty pipeline.
UNESS there is a way to say “remove” the variable setting from the job that is triggering the child pipeline. Is there, as that would make my life so much easier?