What are we trying to do:
We want to have pipeline templates defined in one templates repository globally for the company. These templates may then be used in a different CI repository to manage all our team’s CI pipelines. Finally, in project repositories, we use these pipelines to deploy (infrastructure as) code.
We want to move from stageful to stageless pipelines. For some time, this might mean mixing stageless and stageful jobs in one CI pipeline like in this case.
The lineage goes like this:
terraform.src.yml # This file defines the basic hidden jobs without relations terraform.yml # Visible jobs that extend the src jobs and add relations
sandbox.yml # This file contains all the CI jobs for the specific project
CI/CD configuration file is sourced from CI repository
.image: &image image: $CI_REGISTRY/platform/runner/terraform:latest .tf_init: &tf_init - cd . - terraform init .cache: &cache key: files: - versions.tf paths: - ./.terraform/providers/ - ./.terraform.lock.hcl ###### Hidden jobs (to be extended) ###### .tf_init_cache: <<: *image cache: <<: *cache script: - *tf_init rules: - changes: - "**/versions.tf" .tf_plan: <<: *image cache: <<: *cache policy: pull script: - *tf_init - terraform plan -out default.tf_plan artifacts: paths: - default.tf_plan .tf_apply: <<: *image cache: <<: *cache policy: pull script: - *tf_init - terraform apply -auto-approve default.tf_plan rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH .except_schedules_pipelines: rules: - if: $CI_PIPELINE_SOURCE == "schedule" when: never - if: $CI_PIPELINE_SOURCE == "pipeline" when: never
Jobs without relations in the previous file are extended with relations in this one.
include: - https://ci-files.example.com/infrastructure/terraform.src.v1.yml tf_init_cache: extends: - .tf_init_cache needs:  tf_plan: extends: - .tf_plan needs: - tf_init_cache tf_apply: extends: - .tf_apply needs: - tf_plan allow_failure: false
Now we are actually defining the whole pipeline in CI repository
stages: - lint - apply - k8s-system - k8s include: - "https://ci-files.example.com/infrastructure/terraform.v1.yml" tf plan: extends: - tf_plan - .except_schedules_pipelines tf apply: extends: - tf_apply - .except_schedules_pipelines rules: # allow manual apply also from powersuer/* branches. Data team does a lot of testing - if: $CI_COMMIT_BRANCH =~ /poweruser\/\S+/ when: manual # some other jobs using the above stages
What we expect:
Pipeline runs without problems, in stage view all jobs without stage are executed in
Dependencies are defined and the jobs in
test run according to the defined order
What actually happens:
We see the following error in our project repository:
infra validator job: chosen stage does not exist; available stages are .pre, lint, apply, k8s-system, k8s, .post
GitLab: 15.0 Ultimate, self-managed