Merge Request (feature branch into dev) not Picking up ENV Variables

Working on Proof of Concept (POC) with a very basic/simple pipeline flow: feature → dev → qa → uat → production.

The Problem:
Merging a feature branch into dev runs as detached (which, if I understand correctly, isn’t ideal) and (I think) because it is detached, the pipeline can’t read the env vars (to access AWS).

But, when I merge dev branch into qa, the MR and pipeline run successfully and not detached

The Question

1. What am I doing wrong / how do I merge a feature branch into dev (and start the pipeline correctly).

Details - gitlab-ci.yml
The feature branch doesn’t have a stage or job in the pipeline. Once it is ready, I want to merge the feature into dev, which should then kick off the pipeline.

The below script works (does not run detached) if I merge from dev into qa:

stages:
  - dev
  - qa

dev-jobs:
  stage: dev
  rules: 
    - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "dev" || $CI_COMMIT_BRANCH == "dev"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never 
  before_script: 
    - # omitted for clarity 
  script: 
    - serverless deploy --stage dev --verbose   
  environment: 
    name: dev/$CI_COMMIT_REF_NAME

qa-job:
  stage: qa
  rules: 
    - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "qa"'    
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never # do not run a commit push if there is an open merge request
    - if: "$CI_COMMIT_BRANCH == 'qa'"
  before_script: 
    - # script things that work on push to dev
  script:
    - serverless deploy --stage qa --verbose 
  environment: 
    name: qa/$CI_COMMIT_REF_NAME

What troubleshooting steps have you already taken?
I protected the feature branch - it wasn’t initially nor do I think it should be?

I reviewed:
Pipelines for MR

Workflow Rules

AWS ENV Vars Not Accessed but is 2 years old and not similar in execution.

I’ve looked for merging feature branches into dev/main but haven’t found anything that seems similar.

Welcome @brad.beggs

I think you probably want this, to solve the detatched pipeline issue:

workflow:
    rules:
        - if: $CI_MERGE_REQUEST_ID
          when: never
        - when: always

See here for docs.

In your CI env vars (on the /settings/ci_cd page of your project, see Variables), you need to make sure that the AWS env vars you need are not marked protected. This will ensure that the variables will be available on all branches, not just protected branches.

It would be wise to mask the variables, if possible, then they won’t leak into your pipeline logs.

Those two things might solve your issues, but if not, please do post more details.

Hi @snim2 and thanks for the welcome!

The workflow rules above don’t prevent a detachment when placed a file top.

But, changing the variables from protected to not protected (but masked) does allow a merge.

Not directly related to the problem, but related to my understanding, I don’t follow the case change from when: never to when: always. I’ve not found a explanation of how/why such a sequence works. I read it as: if a merge_request_id then never run and always apply this rule. I suspect I"m wrong though.

I appreciate the suggestions,
Brad

Hi @brad.beggs

The reason for that rule, is that by default, when you push a new commit to a branch GitLab will start a pipeline for the branch. When you push a new commit to a branch with an associated merge request, GitLab will start a pipeline for the merge request.

So, without the workflow rules you will get two pipelines for each push to a branch with an associated MR.

There are two templates available for tweaking this, one gives you MR pipelines only:

workflow:
  rules:
    - if: $CI_MERGE_REQUEST_IID
    - if: $CI_COMMIT_TAG
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

and one gives you branch pipelines only:

workflow:
  rules:
    - if: $CI_COMMIT_TAG
    - if: $CI_COMMIT_BRANCH

You can find the templates here.

By default, if a rule does not have a when key, GitLab will infer when: on_success. Personally, I’d prefer to write that in because I think it’s easier for new developers if each rule is self-documenting.

You might want to play with these rules and see where you get…

The below simple/basic pipeline is what works to avoid detached and to avoid duplicate pipelines running at the same time:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'.   
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push"' 
      when: never
    - if: '$CI_COMMIT_BRANCH'

stages:
  - dev
  - qa
  - uat
  - prod

dev-jobs:
  stage: dev
  rules: 
    - if: '$CI_COMMIT_BRANCH == "dev"'
  before_script: 
    - npm ci --cache .npm --prefer-offline --progress=false
    - npm install -g serverless
    - npm --version 
  script: 
    - serverless deploy --stage dev --verbose   
  environment: 
    name: dev/$CI_COMMIT_REF_NAME

# other stages/jobs removed but resemble the above dev job for now

How I think The Above Works

This took me a while to wrap my head around (overall structure of the workflow rules, stages, jobs, job rules.

Is my understanding correct the first if CI_Pipeline_Source == "merge_request_event" runs a merge based on a matching CI_Commit_Branch because the 3rd if CI_Commit_Branch? If there was no third if, the merge wouldn’t know what to run.

I understand what the 2nd if in the workflow: rules: does (don’t run pipeline on a branch if an open merge exists on a push).

So, if you don’t have a when clause in your rule, the default is on_success. If the job does not have a preceding stage in the pipeline, then on_sucess defaults to true.

So your workflow is the equivalent of:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: on_success
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push"' 
      when: never
    - if: '$CI_COMMIT_BRANCH'
      when: on_success

and I think it’s probably a bit more complicated than you need.

The template for merge request pipelines (from a link in my previous answer) is this:

workflow:
  rules:
    - if: $CI_MERGE_REQUEST_IID
    - if: $CI_COMMIT_TAG
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

which is the equivalent of:

workflow:
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: on_success
    - if: $CI_COMMIT_TAG
      when: on_success
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: on_success

In this example, the if: $CI_COMMIT_TAG clause is only necessary if you also want pipelines for Git tags.

The last clause if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ensures that once an MR has been merged into the default branch, a new pipeline is run on the merged code.