Final Stages/Jobs Not Being Created

Describe your question in as much detail as possible:
I’ve got a set of ci.yml files included in my .gitlab-ci.yml file that specify which jobs to run based on which branch is committed against (more for organization/ease of consumption) than anything else.
These stages/jobs have been running successfully for about 9 months with no issues.
Yesterday, I added a final stage (mr) and placed a job in that stage using a known good/thoroughly tested python script. (that runs as intended when run manually)

During the first test, all 4 stages (and included jobs) were created and run sucessfully.
After spreading the changes to the per-branch ci files, the final stage/job stopped being created when an MR is merged, and I cannot figure out why.

What I’m seeing:
Jobs in the last stage of my gitlab-ci.yml aren’t being created

What I expect to see:
all 4 stages/job created.

my .gitlab-ci.yml:

stages:
  - validate
  - plan
  - push
  - mr

default:
  image:
    name: this points to a valid internal docker image that works
    entrypoint:
      - "/usr/bin/env"
      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

include:
  - '/.common-ci/common_vars.yml'
  - '/development/development-ci.yml'
#__append-new-teams-below__#
  - '/.team-ci/dcs.yml'
  - '/.team-ci/cdn.yml'
  - '/.team-ci/core-infrastructure.yml'
  - '/.team-ci/eo.yml'
  - '/.team-ci/voice-services.yml'
  - '/.team-ci/netsec.yml'
  - '/.team-ci/ire.yml'
  - '/.team-ci/io.yml'

Sample included yml file that isn’t working: (ire.yml)

.infrastructure-dev-jwn-variables: &infrastructure-dev-jwn-vars

  variables:
    NEW_RELIC_API_KEY: "$INFRA_API_KEY_DEV"
    NEW_RELIC_ACCOUNT_ID: "$INFRA_ID_DEV"
    WORKDIR: $CI_PROJECT_DIR/infrastructure-teams/ire/infrastructure-dev-jwn
    PLAN: infrastructure-dev-jwn.tfplan

.infrastructure-jwn-variables: &infrastructure-jwn-vars

  variables:
    NEW_RELIC_API_KEY: "$INFRA_API_KEY_PROD"
    NEW_RELIC_ACCOUNT_ID: "$INFRA_ID_PROD"
    WORKDIR: $CI_PROJECT_DIR/infrastructure-teams/ire/infrastructure-jwn
    PLAN: infrastructure-jwn.tfplan

.noc-jwn-variables: &noc-jwn-vars

  variables:
    NEW_RELIC_API_KEY: "$NOC_API_KEY_PROD"
    NEW_RELIC_ACCOUNT_ID: "$NOC_ID_PROD"
    WORKDIR: $CI_PROJECT_DIR/infrastructure-teams/ire/noc-jwn
    PLAN: noc-jwn.tfplan

.ire-dev-jwn-variables: &ire-dev-jwn-vars

 variables:
   NEW_RELIC_API_KEY: "$IRE_API_KEY_DEV"
   NEW_RELIC_ACCOUNT_ID: "$IRE_ID_DEV"
   WORKDIR: $CI_PROJECT_DIR/infrastructure-teams/ire/ire-dev-jwn
   PLAN: ire-dev-jwn.tfplan

.ire-jwn-variables: &ire-jwn-vars

 variables:
   NEW_RELIC_API_KEY: "$IRE_API_KEY_PROD"
   NEW_RELIC_ACCOUNT_ID: "$IRE_ID_PROD"
   WORKDIR: $CI_PROJECT_DIR/infrastructure-teams/ire/ire-jwn
   PLAN: ire-jwn.tfplan

.merge-to-main-variables: &merge-to-main-vars
  variables:
    MERGE_APPROVER: $INFRASTRUCTURE_APPROVALS
    MERGE_CREATOR: $INFRASTRUCTURE_MERGES

validate_ire-dev-jwn:

  stage: validate
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform validate
  only:
   - ire
  
plan_ire-dev-jwn:

  stage: plan
  <<: *ire-dev-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  
  script:
    - terraform plan -out=$WORKDIR/$PLAN
  artifacts:
    name: ire_dev_plan
    paths:
      - $WORKDIR/$PLAN
  only:
   - ire

deploy_ire-dev-jwn:

  stage: push
  <<: *ire-dev-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform apply -input=false $WORKDIR/$PLAN
  allow_failure: false
  only:
   - ire

validate_ire-jwn:

  stage: validate
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform validate
  only:
   - ire

plan_ire-jwn:

  stage: plan
  <<: *ire-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform plan -out=$WORKDIR/$PLAN
  artifacts:
    name: ire_plan
    paths:
      - $WORKDIR/$PLAN
  only:
   - ire

deploy_ire-jwn:

  stage: push
  <<: *ire-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform apply -input=false $WORKDIR/$PLAN
  allow_failure: false
  only:
    - ire

validate_ire-infrastructure-dev-jwn:

  stage: validate
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform validate
  only:
   - ire
 
plan_ire-infrastructure-dev-jwn:

  stage: plan
  <<: *infrastructure-dev-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init

  script:
    - terraform plan -out=$WORKDIR/$PLAN
  artifacts:
    name: ire_infra_dev_plan
    paths:
      - $WORKDIR/$PLAN
  only:
   - ire

deploy_ire-infrastructure-dev-jwn:

  stage: push
  <<: *infrastructure-dev-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform apply -input=false $WORKDIR/$PLAN
  allow_failure: false
  only:
    - ire 

validate_ire-infrastructure-jwn:

  stage: validate
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform validate
  only:
   - ire
 
plan_ire-infrastructure-jwn:

  stage: plan
  <<: *infrastructure-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform plan -out=$WORKDIR/$PLAN
  artifacts:
    name: ire_infra_plan
    paths:
      - $WORKDIR/$PLAN
  only:
   - ire

deploy_ire-infrastructure-jwn:

  stage: push
  <<: *infrastructure-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform apply -input=false $WORKDIR/$PLAN
  allow_failure: false
  only:
    - ire 

validate_ire-noc-jwn:

  stage: validate
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform validate
  only:
   - ire
 
plan_ire-noc-jwn:

  stage: plan
  <<: *noc-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform plan -out=$WORKDIR/$PLAN
  artifacts:
    name: ire_noc_plan
    paths:
      - $WORKDIR/$PLAN
  only:
   - ire

deploy_ire-noc-jwn:

  stage: push
  <<: *noc-jwn-vars
  before_script:
    - cd $WORKDIR
    - creds-helper -aws
    - terraform init
  script:
    - terraform apply -input=false $WORKDIR/$PLAN
  allow_failure: false
  only:
   - ire

post_deploy_mrs:

  stage: mr
  <<: *merge-to-main-vars
  image:
    name: points to a valid internal docker image (different from the image in the main gitlab-ci.yml)
    entrypoint:
      - "/usr/bin/env"
      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" 
  before_script:
    - pip install virtualenv
    - virtualenv venv
    - . venv/bin/activate
    - python -V
    - pip install python-gitlab
  script:
    - python libraries/merge_bot.py ire
  allow_failure: false
  only:
    - ire

The final stage/script (merge_bot.py) automatically creates MRs from the ire branch and pushes them to test and finally main. We do not allow pushes/merges directly to main, and instead use automation to merge branches “up” to the test branch and from the test branch to main (default)

Hi @flyingcircusmonkey

what is the MR target branch? The post_deploy_mrs has rule to run only on ire branch so if you don’t have a similar job for MR target branch it won’t run there after MR is merged.

The MR target branch is ire, the merge_bot.py takes over and merges to test and then main. This (at least to me) should run on every merge in the IRE branch, and once it runs, will then be pushed by merge_bot.py to test and main.

It’s overly complex because we allow different teams to manage “their own” branches, so there are identical jobs in a number of branches that are also run based on the branch name (each team using this pipeline has their own named branch)

It’s a kludge so we can work around some discrepancies
in the way different teams operate.

This isn’t an answer to your question, but it might be useful for you to know. marge-bot is a tool that will do the same job as your merge_bot.py but does not require any changes to your .gitlab-ci.yml configs. You can configure marge-bot to run on MRs whose source or target branches match a regex, if you need to.

Thanks!!
I wrote merge_bot as a “learn to do something useful” exercise in Python, and while it works flawlessly, trying to wedge it into an existing flow has been a pain