Rules for a scheduled pipeline do not work as expected

GitLab CI/CD
,
1

Rules for a scheduled job does not work as expected

Hello,
We have set our CI to create a pipeline on a push to a branch and set it to manual. There is also a configured schedule to run every hour. We want the scheduled pipeline to be started only if there is a change in the code since the last time the pipeline was executed. The way we tried to implement this is by comparing the commit sha.
The way the CI is configured is to use rules

rules:
- if: $CI_COMMIT_BRANCH == "Active_Development" && $CI_PIPELINE_SOURCE == "push"
  when: manual
- if: $CI_COMMIT_SHA != $LAST_SHA && $CI_COMMIT_BRANCH == "Active_Development" && $CI_PIPELINE_SOURCE == "schedule"
  when: on_success
- when: never

$LAST_SHA variable is extracted from the pipeline API → curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB/api/v4/projects/54670/pipelines?per_page=3&&sort=desc" |jq -c '.[2] | .sha' | sed -e 's/^"//' -e 's/"$//'

This rule is added to all jobs and stages.

The problem is that the schedule is creating a pipeline every hour no matter that the $CI_COMMIT_SHA is equal to the value of $LAST_SHA

  • Self-managed GitLab

Example:

pipeline
pipeline2108Ă—871 138 KB

The idea is the following:

  • Pipeline #1950986 is started by the scheduler and executed successfully
  • Code to Actvive_Development branch has been pushed and pipeline #1951087 has been created and set to manual
  • Pipeline #1951088 is started by the scheduler and executed successfully (compare the $CI_COMMIT_SHA - 4bc27a6 with the $LAST_SHA - 7f2910b9 and if different create the pipeline)
  • Pipeline #1951110 is started and it fails because of the same docker manifest ( it should not start as the $CI_COMMIT_SHA - 4bc276a6 is the same as $LAST_SHA - 4bc276a6 )

Can you give an advise how to make this work

2

Hi @dshatarov,
could you please show where and how do you set the $LAST_SHA variable?

3

Hi @balonik

I am setting the LAST_SHA variable in the before_script section in each job I am applying the rules:

.build:
  stage: build
  variables:
    CI_DEBUG_TRACE: "true"
  before_script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}
    - LAST_SHA=$(curl -s --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects/54670/pipelines?per_page=3&&sort=desc" |jq -c '.[2] | .sha' | sed -e 's/^"//' -e 's/"$//')
    - echo $CI_COMMIT_SHA
    - echo $LAST_SHA
  after_script:
    - docker logout ${CI_REGISTRY}

build:dev:sch-api:
  stage: build
  extends: .build
  script:
    - docker build -f ./Dockerfile-DECC -t ${CI_REGISTRY}/${CI_REG_IMAGE_DEV}:${CI_COMMIT_SHORT_SHA} .
    - docker push ${CI_REGISTRY}/${CI_REG_IMAGE_DEV}:${CI_COMMIT_SHORT_SHA}
    - docker rmi ${CI_REGISTRY}/${CI_REG_IMAGE_DEV}:${CI_COMMIT_SHORT_SHA}

  rules:

    - if: $CI_COMMIT_BRANCH == "Active_Development" && $CI_PIPELINE_SOURCE == "push"
      when: manual
    - if: $CI_COMMIT_SHA != $LAST_SHA && $CI_COMMIT_BRANCH == "Active_Development" && $CI_PIPELINE_SOURCE == "schedule"
      when: on_success
    - when: never

With the echo I verify the value is extracted correctly from the API. However, it seems the comparison between both commit SHAs does not work.

4

rules are evaluated before any before_script is executed, even if that woudn’t be the case environment variables that you define in before_script, script or after_script are only available in context of that shell. They are not propagated to GitLab Runner and cannot be used anywhere else.

You can look at another post with similar use-case: How to launch a job at 11pm only when there was a new push - #5 by balonik

5

Hi @balonik
I forgot to update the thread but I think it my be useful for other users. I managed to achieve the desired outcome by adding an additional stage “schedule”. There I run several Gtilab API calls and based on the results the pipeline continues:

schedule:business-api:
  stage: schedule
  script:
    - LAST_SHA=$(curl -s --header "PRIVATE-TOKEN:$GITLAB_API" "$GITLAB_URL/api/v4/projects/$CI_PROJECT_ID/pipelines?per_page=5&&sort=desc" |jq -c '.[] | select(.ref == "Active_Development")' | jq '.sha' | sed '3q;d' | sed -e 's/^"//' -e 's/"$//')
    - LAST_PIPE=$(curl -s --header "PRIVATE-TOKEN:$GITLAB_API" "$GITLAB_URL/api/v4/projects/$CI_PROJECT_ID/pipelines?per_page=1&&sort=desc" |jq -c '.[0] | .id')
    - > 
      if [ "$LAST_SHA" == "$CI_COMMIT_SHA" ]; then 
         echo "There are no changes since last run"
         curl --request POST --header "PRIVATE-TOKEN:$GITLAB_API" "$GITLAB_URL/api/v4/projects/$CI_PROJECT_ID/pipelines/$LAST_PIPE/cancel"
         curl --header "PRIVATE-TOKEN:$GITLAB_API" --request "DELETE" "$GITLAB_URL/api/v4/projects/$CI_PROJECT_ID/pipelines/$LAST_PIPE"
      fi   
  only: 
    - schedules
.build:
  stage: build
  before_script:
    - docker login -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD} ${CI_REGISTRY}
  after_script:
    - docker logout ${CI_REGISTRY}
build:dev:pub-work:
  stage: build
  extends: .build
  script:
    - docker build -f ./Dockerfile-DECC -t ${CI_REGISTRY}/${CI_PUB_IMAGE_DEV}:${CI_COMMIT_SHORT_SHA} .
    - docker push ${CI_REGISTRY}/${CI_PUB_IMAGE_DEV}:${CI_COMMIT_SHORT_SHA}
    - docker rmi ${CI_REGISTRY}/${CI_PUB_IMAGE_DEV}:${CI_COMMIT_SHORT_SHA}
  rules:
    - if: $CI_COMMIT_BRANCH == "Active_Development" && $CI_PIPELINE_SOURCE == "push"
      when: manual
    - if: $CI_COMMIT_BRANCH == "Active_Development" && $CI_PIPELINE_SOURCE == "schedule"
      when: on_success
    - when: never
1 Like