Job would create infinitely looping pipeline

Problem

We use a multi-repo/superproject setup with several services as git submodules. During a deploy, a job is trigged on the app (superproject) repo which then triggers the “build” (and later “deploy”) job on the child repositories. This setup has been in use for over a year and has been working well for us.

Today I noticed that this process is broken (for at least one of our projects) because Gitlab CI kills off the trigger job (build/server) within the superproject because “Job would create infinitely looping pipeline”. This wasn’t the case in the past and I do not see why it would (think that it would) create an infinitely looping pipeline.

We are on the SaaS platform, so perhaps Gitlab 14.0 was rolled out? However I couldn’t see anything about looping pipelines in the blog post. Also, the Gitlab CI Runner commit is still the same as a few days ago.

Config

Main App / Superrepo

  • App Repo starts pipeline:

    export::ref/server -> trigger::subprojects/production
    

    This exports the references for the submodules and triggers a job for each of those that have changed (and therefor needs a build + deploy).

  • The trigger job kicks off on the main repo / superrepo:

    build/server -> deploy/server
    

    The build/XXXX and subsequently deploy/XXXX jobs are triggered within the child project.

Relevant config of the main (“app”) project / superproject:

##############
## REF JOBS ##
##############
.export::ref:
  stage: subproject refs
  image:
    name: alpine/git:latest
    entrypoint: [""]
  except:
    # prevent loop of triggering own pipeline again.
    - triggers
    - pipelines
  script:
    - git rev-parse @:services/$(echo "$PROJECT_NAME" | tr A-Z a-z | tr _ -) >> ${PROJECT_NAME}_COMMIT_SHA
  artifacts:
    paths:
      - ${PROJECT_NAME}_COMMIT_SHA

export::ref/server:
  extends: .export::ref
  only:
    refs:
      - master
    changes:
      - services/server
  variables:
    PROJECT_NAME: SERVER

##################
## TRIGGER JOBS ##
##################
.base::trigger::subprojects:
  image: registry.gitlab.com/PROJECT/trigger-pipeline:latest
  stage: trigger
  variables:
    PROJECTS: 'SERVER'
    RT_PROJECT_ID: XXXXXXXX
  except:
    # prevent loop of triggering own pipeline again.
    - triggers
    - pipelines
  script:
    - |
      construct_env_args() {
        for PROJECT in ${PROJECTS//,/ }
        do
          # call your procedure/other scripts here below
          if [[ -f "${PROJECT}_COMMIT_SHA" ]]; then
              export ${PROJECT}_COMMIT_SHA=`cat ${PROJECT}_COMMIT_SHA`
              tmpvar="${PROJECT}_COMMIT_SHA"
              [[ ! -z ${!tmpvar} ]] && env_options+=( --env "${PROJECT}_COMMIT_SHA"="${!tmpvar}")
           fi
        done
      }
    - construct_env_args
    - trigger --detached -p ${CI_JOB_TOKEN} -t ${RT_REF} "${env_options[@]}" --env ENVIRONMENT=${CI_ENVIRONMENT_SLUG} ${RT_PROJECT_ID}
  dependencies:
    - export::ref/server

trigger::subprojects/production:
  extends: .base::trigger::subprojects
  only:
    refs:
      - master
    changes:
      - services/server
  environment:
    name: production
  variables:
    RT_REF: 'master'
    
#######################
## BASE BUILD/DEPLOY ##
#######################
.base::build:
  stage: build
  inherit:
    variables: false
  variables:
    ENVIRONMENT: $ENVIRONMENT
    APP_STAGE: 'build'

.trigger-server: &trigger-server
  trigger:
    project: PROJECT/services/server
    branch: master
    strategy: depend
  only:
    variables:
      - $SERVER_COMMIT_SHA
  variables:
    COMMIT_SHA: $SERVER_COMMIT_SHA

###########
## BUILD ##
###########
build/server:
  extends: .base::build
  <<: *trigger-server

############
## DEPLOY ##
############
deploy/server:
  extends: .base::deploy
  <<: *trigger-server

Child project (triggered by app)

Relavant config of the child project:

build::docker/app:
  extends:
    - .base
    - .docker/build-aws # see includes
  rules:
    - if: '$APP_STAGE == "build" && $COMMIT_SHA'
  variables:
    RT_DOCKER_TARGET: app
    RT_DOCKER_IMAGE_TAG: $COMMIT_SHA
    RT_DOCKER_TAG_AS_LATEST: "true"

## Includes
.base:
  environment:
    name: production
  dependencies:
    - aws/assume-role
  before_script:
    - chamber export ...............

.variables:
  RT_DOCKER_TAG_AS_LATEST: 'true'
  RT_DOCKERFILE: 'Dockerfile'

.docker/build-aws:
  stage: build
  image: registry.gitlab.com/PROJECT/build-container:latest
  services:
    - ekino/ci-dind:latest
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_HOST: 'tcp://ekino__ci-dind:2375'
  script:
    # Login to Gitlab Registry
    # Check if all required variables are set.
    # Login to the AWS ECR Registry
    # Pull latest image to use as cache during build.
    # Build docker image
    # Push image to AWS ECR Registry

Can anyone point me in the direction as how to solve / troubleshoot this? The error message from the pipelines isn’t very helpful in drilling down as to “Why”.

1 Like

I was hoping that moving away from only and except in favor of rules would fix the issue, but unfortunately it is still failing.

It seems to be a new “feature”, but how can I properly do the triggering instead?

1 Like

Having the same problem here and i already had rules applied.

Did anyone have a solution ?

I have reported this to Gitlab directly, unfortunately nobody has replied to the issue yet.

Quite problematic because it is affecting our entire build/deploy pipeline for all of the projects within our company and we have had to invest a lot of time on it investigating and looking for alternatives.