Is there a bug using the pipeline trigger API ? (Two different outputs of git checkout from the same git hash)

GitLab CI/CD
, , ,
1

I have developed a tool which allows users to run predefined pipeline in Gitlab-CI, and I run the pipeline via API. Inside the pipeline, there is some logic going on, mainly Ansible related.

At some point, there is a commit. Lately, I have face some problems because the commit started to fail in some cases, because a git rebase was needed. I improved the logic, wrote a script that I call inside the pipeline script.

Now, 90% of my pipelines works, and the others 10% fail, because the script is missing. All the pipelines are triggered at the same point in time (a few milliseconds delay), so it should start on the same commit, using the same repository state.

Example : Runner that succeeded :ok:

Running with gitlab-runner 15.11.0 (436955cb)
  on gitlab-runner-node-201 GKC5Dcxe, system ID: s_7bbf7487f540
[...]
section_start:1689279186:get_sources
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/ansible-central/ansible-command-center/.git/
Checking out a1387351 as detached HEAD (ref is disable-rebase)...
[...]
$ git log -1 --oneline
a138735 🤔 Ajout debug
[...]
$ git config --global user.name "${author_name}"
$ git config --global user.email "${author_email}"
$ git config pull.rebase true
$ git checkout ${CI_COMMIT_REF_NAME}
Previous HEAD position was a138735 🤔 Ajout debug
Switched to branch 'disable-rebase'
Your branch is behind 'origin/disable-rebase' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
$ git branch -u origin/${CI_COMMIT_REF_NAME}
branch 'disable-rebase' set up to track 'origin/disable-rebase'.

Example : Runner that failed :x:

Running with gitlab-runner 15.11.0 (436955cb)
  on gitlab-runner-node-203 xsX26gLC, system ID: s_7bbf7487f540
[...]
section_start:1689279121:get_sources
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/ansible-central/ansible-command-center/.git/
Checking out a1387351 as detached HEAD (ref is disable-rebase)...
[...]
$ git log -1 --oneline
a138735 🤔 Ajout debug
[...]
$ git config --global user.name "${author_name}"
$ git config --global user.email "${author_email}"
$ git config pull.rebase true
$ git checkout ${CI_COMMIT_REF_NAME}
Previous HEAD position was a138735 🤔 Ajout debug
Switched to branch 'disable-rebase'
Your branch and 'origin/disable-rebase' have diverged,
and have 1 and 50 different commits each, respectively.
[...]
/bin/bash: line 235: ./juno/utils/commit.sh: No such file or directory

So I don’t understand :

  1. The two runners were triggered on the same time (1689279186 and 1689279121)
  2. The git commit was the same : a138735
  3. When I run the command git checkout ${CI_COMMIT_REF_NAME} I get two different results :
Previous HEAD position was a138735 🤔 Ajout debug
Switched to branch 'disable-rebase'
Your branch is behind 'origin/disable-rebase' by 2 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Previous HEAD position was a138735 🤔 Ajout debug
Switched to branch 'disable-rebase'
Your branch and 'origin/disable-rebase' have diverged,
and have 1 and 50 different commits each, respectively.

How can the two commands, from the same state of the repo gives two different outputs ?

2

Are you using shell executor?

3

No, all runners use docker executor.

4

Ah ok. After a quick look this looks to me you are not expecting that the GIT repo is cached on the Runner.

Be aware that Docker executor is caching GIT files locally so if you alter them within the job it is re-used in further jobs. You need to be aware especially if you do git commands within the jobs. You can see it in the job output, Reinitialized existing Git repository in... means it’s using locally cached GIT clone; Initialized empty Git repository in... means it’s a fresh clone.

So let’s say I have completely empty repository except .gitlab-ci.yml which I need.

(omitting unimportant definitions)

stages:
  - create
  - read

create file:
  stage: create
  script:
    - git config --global user.email "you@example.com"
    - git config --global user.name "Your Name"
    - git checkout -b test
    - echo test > test.txt
    - git add test.txt
    - git commit -m 'add test'

read file:
  stage: read
  script:
    - git checkout test
    - cat test.txt

If the jobs run on the same machine/runner since the repository is cached I can read file test.txt even without it being pushed to remote, ever.

create file job output snippet:

...
Initialized empty Git repository in /builds/tests/ghost-file/.git/
...
$ git config --global user.email "you@example.com"
$ git config --global user.name "Your Name"
$ git checkout -b test
Switched to a new branch 'test'
$ echo test > test.txt
$ git add test.txt
$ git commit -m 'add test'
[test 35ad5b9] add test
 1 file changed, 1 insertion(+)
 create mode 100644 test.txt
Cleaning up project directory and file based variables
00:01
Job succeeded

read file job output snippet:

...
Reinitialized existing Git repository in /builds/tests/ghost-file/.git/
...
$ git checkout test
Previous HEAD position was 816490d test
Switched to branch 'test'
$ cat test.txt
test
Cleaning up project directory and file based variables
00:00
Job succeeded

:magic_wand:

I hope this helps you understand whats going on.

5

Thank you for your insights. I thought this could be linked to cached data, soI added the cache directive to the empty array, thinking it would disabled it, as show in documentation :


juno:deploy:
  stage: "juno:deploy"
  cache: []
  tags:
    - juno
  variables:
    LANG: C.UTF-8
    LC_ALL: C.UTF-8
    ...

Be aware that Docker executor is caching GIT files locally

Does this behaviour can be altered by the cache directive ?

6

No, this does not apply to it.

7

To solve this thread, here is how I handled the problem :

  1. Add the empty cache directive to the job cache: []
  2. Implement a try catch for the rebase :

commit.sh

#!/bin/bash

source $(pwd)/trycatch.sh

export REBASE_EXCEPTION=100

# File edition logic removed

try
(
    git pull --rebase origin ${CI_COMMIT_REF_NAME} || throw $REBASE_EXCEPTION
)
catch || {
    # now you can handle
    case $ex_code in
        $REBASE_EXCEPTION)
            echo "💥 Rebase error, retrying"
            git rebase --abort
            git pull --rebase origin ${CI_COMMIT_REF_NAME}
        ;;
    esac
}

git push --set-upstream origin ${CI_COMMIT_REF_NAME}
echo "✨ Succesfully commited"

try-catch.sh

#!/bin/bash

function try()
{
    [[ $- = *e* ]]; SAVED_OPT_E=$?
    set +e
}

function throw()
{
    exit $1
}

function catch()
{
    export ex_code=$?
    (( $SAVED_OPT_E )) && set +e
    return $ex_code
}

function throwErrors()
{
    set -e
}

function ignoreErrors()
{
    set +e
}