Intermittent Failure to Clone Submodules in CI Jobs
Hello! I am seeing an ongoing issue where submodules are failing to clone during the initial stages of CI job. This is occurring across multiple projects. An authentication failure is reported in the CI log (see below). The failures occur ~30% of the time. The issue can be “fixed” by simply re-running the CI job. Sometimes it takes multiple re-runs until the submodules are cloned successfully. It is the same error each time. The submodules can be cloned locally on several different systems by different GitLab users without an issue. Up until recently, this was entirely a non-issue with the same submodules. I’m not 100% sure what has changed since then.
I have reviewed the GitLab logs and the Runner logs during these failures, but I haven’t found any major error or clues.
The .gitlab-ci.yml file has, among the actual job definitions (which run without issue once the submodules are cloned), this variable set:
GIT_SUBMODULE_STRATEGY: recursive
There are no other submodule-related vars or commands in the .gitlab-ci.yml.
This is on a self-managed GitLab version 15.6 and Docker runner version 15.6.
Does anyone have ideas why this is happening and how to resolve it? Thanks!
CI job log:
Running with gitlab-runner 15.6.1 (133d7e76)
on docker-runner-2a Qhwo4AhD
Preparing the "docker" executor 00:01
Using Docker executor with image gitlab.mycompany.com:4567/tools/build-image:latest ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image gitlab.mycompany.com:4567/tools/build-image:latest ...
Using docker image sha256:df34ccaebb31ff7a08f8b5a661a1e181dbd602420fa11b252603c39712f678e2 for gitlab.mycompany.com:4567/tools/build-image:latest with digest gitlab.mycompany.com:4567/tools/build-image@sha256:66d2a312e28f1820116064d1a49fc6d053351e32e265985cc52d7a217106011a ...
Preparing environment 00:00
Running on runner-qhwo4ahd-project-869-concurrent-0 via docker-runner-2.mycompany.com...
Getting source from Git repository 00:01
Fetching changes...
Reinitialized existing Git repository in /builds/test-build/my-project-1/.git/
Checking out 3bad1de6 as master...
Updating/initializing submodules recursively...
Synchronizing submodule url for 'submodule-one'
Synchronizing submodule url for 'submodule-two'
Synchronizing submodule url for 'submodule-three'
Synchronizing submodule url for 'submodule-four'
Submodule path 'submodule-one': checked out '5d777b83c78d0a40045385e0e94e3d465b8d8749'
Submodule path 'submodule-one/common': checked out '0553e3db01c2accf1a058c34204ea9b25a60a58b'
Submodule path 'submodule-one/ci-scripts': checked out '5465196e9e08de138c21854d2c6df0ba0e47c13d'
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.mycompany.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.mycompany.com/development/submodule-two/'
Unable to fetch in submodule path 'submodule-two'; trying to directly fetch 3c8e8254fe6ee0c132511e2c9a6fa41d2b23a9f5:
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.mycompany.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.mycompany.com/development/submodule-two/'
Fetched in submodule path 'submodule-two', but it did not contain 3c8e8254fe6ee0c132511e2c9a6fa41d2b23a9f5. Direct fetching of that commit failed.
Uploading artifacts for failed job 00:01
Uploading artifacts...
WARNING: .build: no matching files. Ensure that the artifact path is relative to the working directory
ERROR: No files to upload
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1
The project’s .gitmodules:
[submodule "submodule-two"]
path = submodule-two
url = ../../development/submodule-two
branch = master
[submodule "submodule-three"]
path = submodule-three
url = ../../development/submodule-three
branch = master
[submodule "submodule-one"]
path = submodule-one
url = ../../development/submodule-one
branch = master
[submodule "submodule-four"]
path = submodule-four
url = ../../development/submodule-four
branch = develop