Submodule fetch is not working on second Pipeline triggered Task

Describe your question in as much detail as possible:
I have this two Repos
A-Role: https://gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role
B-Test: https://gitlab.sub.example.com/main-project/sub-project/sub-sub-project/B-Test

A-role is the Repo of the Submodule i want to checkout in B-Test when the Repo A-role changes.

Workflow:

  • A-Role changes triggers Pipeline on B-Test → works

  • First step of Pipeline B-Test is to update the Submodule and push it to a new branch → works

  • Second Step is to create a Merge Request → fails as it can’t update the Submodule

  • Third Step run a test_config task → works (manually triggered) does update Submodule without a Problem

  • What are you seeing, and how does that differ from what you expect to see?

The Second step of B-Test “create mr” fails to update the Submodule, but it should work like the Task before or after it

This is the Trigger on Repo A-Role

trigger_B-Test:
  stage: trigger
  trigger:
    project: main-project/sub-project/sub-sub-project/B-Test
  rules:
    - if: '$CI_COMMIT_REF_NAME == "main"'

The simplified gitlab-ci.yml on B-Test looks like this:

variables:
 variables:
  #GIT_DEPTH: 1                               # Create a shallow copy
  BOT_NAME: "GitLab Runner Bot"              # Bot's name that appears in the commit log
  BOT_EMAIL: "gitlab-runner-bot@example.net" # Bot's email, not important
  COMMIT_MESSAGE: "CI File Upload"      # Part of the commit message
  GIT_SUBMODULE_STRATEGY: recursive
  GIT_SUBMODULE_FORCE_HTTPS: "true"
  GIT_STRATEGY: clone # clone entire repo instead of reusing workspace
  GIT_DEPTH: 0 # avoid shallow clone to give sonar all the info it needs
  GIT_SUBMODULE_DEPTH : 0

  
stages:
  - test_config
  - merge-request


test config:
  stage: test_config
  interruptible: true
  cache:
    key: apache-$CI_COMMIT_REF_SLUG
    paths:
      - out/
      - ssl/
    policy: pull
  script:
    - export CI_PROJECT_DIR_SAVED=$CI_PROJECT_DIR
    <<snip>>
  allow_failure: false
  rules:
    - if: $CI_COMMIT_MESSAGE !~ /^CI File Upload.*/ && $CI_PIPELINE_SOURCE == 'merge_request_event'

upstream change:
  stage: merge-request
  rules:
      - if: $CI_PIPELINE_SOURCE == "pipeline"
  script:
    - git checkout -b upstream-change || git checkout upstream-change
    - git submodule init
    - git submodule update
    - cd roles/
    - git fetch
    - git pull origin main
    - cd ..
    - git add roles/
    - git commit -m "roles Submodule Updated"
    - git remote add gitlab_origin "https://gitlab-runner:${GIT_PUSH_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}" || true
    - git push gitlab_origin upstream-change

create mr:
  stage: merge-request
  rules:
      - if: $CI_OPEN_MERGE_REQUESTS == null && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
  script:
      - sh open-merge-request.sh        

The .gitmodules

[submodule "roles"]
	path = roles
	url = ../A-role.git

The First Task “upstream change” works fine without any Problems:

Running with gitlab-runner 16.3.1 (f5dfa4d1)
  on gitlab-runner.sub.example.com bxLs67T1, system ID: s_42e67943a6ab
Resolving secrets 00:00
Preparing the "shell" executor 00:00
Using Shell (bash) executor...
Preparing environment 00:00
Running on gitlab-runner.sub.example.com...
Getting source from Git repository 00:02
Fetching changes...
Initialized empty Git repository in /home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/.git/
Created fresh repository.
Checking out 8210ac01 as detached HEAD (ref is main)...
Updating/initializing submodules recursively...
Submodule 'roles' (https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git) registered for path 'roles'
Synchronizing submodule url for 'roles'
Cloning into '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles'...
Submodule path 'roles': checked out '6c8caf2b30964576ca50d34eb7767ddd000da716'
Updated submodules
Entering 'roles'
Executing "step_script" stage of the job script 00:02
$ git checkout -b upstream-change || git checkout upstream-change
Switched to a new branch 'upstream-change'
$ git submodule init
$ git submodule update
$ cd roles/
$ git fetch
$ git pull origin main
From https://gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role
 * branch            main       -> FETCH_HEAD
Updating 6c8caf2..b0911a0
Fast-forward
 apache-reverseproxy/templates/default-ssl.conf.j2 | 2 +-
 apache-reverseproxy/templates/default.conf.j2     | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
$ cd ..
$ git add roles/
$ git commit -m "roles Submodule Updated"
[upstream-change 114acab] roles Submodule Updated
 Committer: GitLab Runner <gitlab-runner@example.com>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:
    git config --global --edit
After doing this, you may fix the identity used for this commit with:
    git commit --amend --reset-author
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git remote add gitlab_origin "https://gitlab-runner:${GIT_PUSH_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}" || true
$ git push gitlab_origin upstream-change
warning: redirecting to https://gitlab.sub.example.com/main-project/sub-project/sub-sub-project/B-Test.git/
remote: 
remote: To create a merge request for upstream-change, visit:        
remote:   https://gitlab.sub.example.com/main-project/sub-project/sub-sub-project/B-Test/-/merge_requests/new?merge_request%5Bsource_branch%5D=upstream-change        
remote: 
To https://gitlab.sub.example.com/main-project/sub-project/sub-sub-project/B-Test
 * [new branch]      upstream-change -> upstream-change
Cleaning up project directory and file based variables 00:00
Job succeeded

The Second Task “create mr” is successfully triggered but fails with this:

Running with gitlab-runner 16.3.1 (f5dfa4d1)

on gitlab-runner.sub.example.com bxLs67T1, system ID: s_42e67943a6ab

Resolving secrets 00:00

Preparing the "shell" executor 00:00

Using Shell (bash) executor...

Preparing environment 00:00

Running on gitlab-runner.sub.example.com...

Getting source from Git repository 00:03

Fetching changes...

Initialized empty Git repository in /home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/.git/

Created fresh repository.

Checking out 114acab1 as detached HEAD (ref is upstream-change)...

Updating/initializing submodules recursively...

Submodule 'roles' (https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git) registered for path 'roles'

Synchronizing submodule url for 'roles'

Cloning into '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles'...

remote: The project you were looking for could not be found or you don't have permission to view it.

fatal: repository 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git/' not found

fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git' into submodule path '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles' failed

Failed to clone 'roles'. Retry scheduled

Cloning into '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles'...

remote: The project you were looking for could not be found or you don't have permission to view it.

fatal: repository 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git/' not found

fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git' into submodule path '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles' failed

Failed to clone 'roles' a second time, aborting

Updating submodules failed. Retrying...

Synchronizing submodule url for 'roles'

Cloning into '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles'...

remote: The project you were looking for could not be found or you don't have permission to view it.

fatal: repository 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git/' not found

fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git' into submodule path '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles' failed

Failed to clone 'roles'. Retry scheduled

Cloning into '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles'...

remote: The project you were looking for could not be found or you don't have permission to view it.

fatal: repository 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git/' not found

fatal: clone of 'https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git' into submodule path '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles' failed

Failed to clone 'roles' a second time, aborting

Cleaning up project directory and file based variables 00:00

ERROR: Job failed: exit status 1

When I create a Merge Request on the B-Test on my own the Task triggers and works:

Running with gitlab-runner 16.3.1 (f5dfa4d1)
  on gitlab-runner.sub.example.com bxLs67T1, system ID: s_42e67943a6ab
Resolving secrets 00:00
Preparing the "shell" executor 00:00
Using Shell (bash) executor...
Preparing environment 00:01
Running on gitlab-runner.sub.example.com...
Getting source from Git repository 00:02
Fetching changes...
Initialized empty Git repository in /home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/.git/
Created fresh repository.
Checking out 114acab1 as detached HEAD (ref is refs/merge-requests/52/head)...
Updating/initializing submodules recursively...
Submodule 'roles' (https://gitlab-ci-token:[MASKED]@gitlab.sub.example.com/main-project/sub-project/sub-sub-project/A-role.git) registered for path 'roles'
Synchronizing submodule url for 'roles'
Cloning into '/home/gitlab-runner/builds/bxLs67T1/0/main-project/sub-project/sub-sub-project/B-Test/roles'...
Submodule path 'roles': checked out 'b0911a0fef9c0859a2ab15076705be3761c49165'
Updated submodules
Entering 'roles'
Restoring cache 00:01
Checking cache for apache-upstream-change-non_protected...
Runtime platform                                    arch=amd64 os=linux pid=24365 revision=f5dfa4d1 version=16.3.1
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script 00:11
$ export CI_PROJECT_DIR_SAVED=$CI_PROJECT_DIR
$ . /etc/apache2/envvars
<<snip>>
  • What version are you on? Are you using self-managed or GitLab.com?

    • GitLab (Hint: /help): self-managed 15.11.11-ee
    • Runner (Hint: /admin/runners): gitlab-runner_16.3.1_amd64
  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
    I enabled the Token Access under CI/CD in the A-Role Project and added the B-Test to it.
    On a Change on the A-role repository I trigger a Pipeline of the B-Test Repository with this:

Thanks for reading this long Post :slight_smile:

Can you check your .gitmodules after the upstream change? My first idea was that somehow during the upstream change the definition of submodule got changed and now contains the CI_JOB_TOKEN of the upstream change job, which is invalid after the job is completed and therefore the second job (trying to pull submodule with expired token in definition) fails. Totally just a guess :smiley:

Good hint, but it does not change.
The only thing that changes between the created branch and the master is the Subproject commit