CI/CD pipeline can't clone submodules

GitLab CI/CD
,
1

Problem to solve

I have a self-managed Gitlab v18.3.1 and gitlab-runner on the same machine (Ubuntu 24).
I can clone the main project and the submodules manually, but when i want to do it via the CI/CD piepeline it don’t work.

I got the following error:

Cloning into ‘/home/gitlab-runner/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga/src/vhdl-library’…
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

fatal: clone of ‘git@192.168.178.122:ingburo/ifen/libraries/vhdl-library.git’ into submodule path ‘/>home/gitlab-runner/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga/src/vhdl-library’ failed
Failed to clone ‘src/vhdl-library’. Retry scheduled

Configuration

variables:
GIT_STRATEGY: clone
GIT_SUBMODULE_STRATEGY: recursive
GIT_SUBMODULE_FORCE_HTTPS: “true”
stages: # List of stages for jobs, and their order of execution

  • build
    build-job: # This job runs in the build stage, which runs first.
    stage: build
    script:
    • echo “Compiling the code…”
    • cd build
    • set -x
    • export XILINXD_LICENSE_FILE=xxx@yyyyyyyyy
    • /home/build/Xilinx/Vivado/2020.2/bin/vivado -mode tcl -source build_npm.tcl -notrace
    • echo “Compile complete.”
      artifacts:
      name: “artifacts-replayer”
      paths:
      • build/build_data_npm/standard_70/replayer_standard_70.bit

The job token permissions are set as :

image
image1133×501 51.6 KB

2

Does anybody have an idea what could be wrong?

3

The submodule is added with SSH (note the git@ in the clone operation) and a static IP address which by default is not trusted by SSH. Likely you’ll also need to setup SSH keys to being able to clone the repository.

I think that CI/CD needs to setup SSH clone beforehand using before_script:

  1. Create SSH keys, add to CI/CD variables and then use them Using SSH keys with GitLab CI/CD | GitLab Docs
  2. SSH host keys scans Using SSH keys with GitLab CI/CD | GitLab Docs

Alternatively, change the Git submodule to use https with access tokens, if that works for the user kicking off the CI/CD pipeline.

4

I thought i forceed it to use https by adding :

GIT_SUBMODULE_FORCE_HTTPS: “true”

I also configured the Job token permissions. See above…

5

Ah, thanks! I did not know that you can actually rewrite submodules from SSH to HTTPS in CI/CD.

So for some reason, the rewrite does not work, looking at

fatal: clone of ‘git@192.168.178.122:ingburo/ifen/libraries/vhdl-library.git’ into submodule path ....failed

Can you try using an FQDN domain instead of the IP address to see whether it works?

Something like this in your .gitsubmodules config:

[submodule "vhdl-library"]
  path = vhdl-library
-  url = git@192.168.178.122:ingburo/ifen/libraries/vhdl-library.git
+  url = git@gitlab.example.com:ingburo/ifen/libraries/vhdl-library.git

If an FQDN does not work, we’ll have to look somewhere else - maybe a bug.

Another question: Which GitLab Runner version is used?

gitlab-runner --version
6

Hi. Thanks for your quick response.
I change the .gitsubmodules entry to:

url = git@Ing-Buro-Server:ingburo/ifen/libraries/vhdl-library.git

using:

hostname --fqdn

to get the correct FQDN, but nothing changed. Same behavior as before.

Gitlab v18.5.0 and gitlab-runner are on the same machine (Ubuntu 24).

GitLab Runner:

Version:      18.5.0
Git revision: bda84871
Git branch:   18-5-stable
GO version:   go1.24.6 X:cacheprog
Built:        2025-10-13T19:20:30Z
OS/Arch:      linux/amd64
7

I feared as much. I did some source code peeks to understand how the rewrite is done. It confirms that both, IP and FQDN should work.

Next steps:

  1. Set the CI_DEBUG_TRACE variable in CI/CD and trace with more details in the job logs
  2. Enable GitLab Runner debug logging and inspect if there are submodule setup details

Details

I’ve peeked into the gitlab-org/gitlab-runner source code and asked GitLab Duo Agentic Chat with a prompt:

I want to clone gitsubmodules in CI/CD using a CI/CD variable override from SSH to HTTPs.

Does this work with FQDN domains only, or are IP addresses supported. I assume a regex is used.

image
image1920×1259 199 KB

image
image1920×1259 200 KB

Agentic Chat comes back analyzing the code in common/build_url_helper.go · main · GitLab.org / gitlab-runner · GitLab and available docs.

This is latest 18.5, so there might be difference in 18.3.1. The feature was added in 15.11 though.

image
image1920×1259 202 KB

8

After doing step 1 (CI_DEBUG_TRACE: “true”)
and step 2: Enable GitLab Runner debug logging

gitlab-runner stop
gitlab-runner --debug run

i get the following output:

Runtime platform                                    arch=amd64 os=linux pid=25449 revision=bda84871 version=18.5.0
Starting multi-runner from /home/mibo/.gitlab-runner/config.toml...  builds=0 max_builds=0
Checking runtime mode                               GOOS=linux uid=1000
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
Usage logger disabled                               builds=0 max_builds=1
Configuration loaded                                builds=0 max_builds=1
listenaddress: ""
sessionserver:
  listenaddress: ""
  advertiseaddress: ""
  sessiontimeout: 1800
labels: {}
concurrent: 1
checkinterval: 0
loglevel: null
logformat: null
user: ""
runners: []
sentrydsn: null
connectionmaxage: null
modtime: 0001-01-01T00:00:00Z
loaded: false
experimental: null
shutdowntimeout: 0
configsaver: null
  builds=0 max_builds=1
Waiting for stop signal                             builds=0 max_builds=1
listen_address not defined, metrics & debug endpoints disabled  builds=0 max_builds=1
[session_server].listen_address not defined, session endpoints disabled  builds=0 max_builds=1
Initializing executor providers                     builds=0 max_builds=1
Feeding runners to channel                          builds=0 max_builds=1
Starting worker                                     builds=0 max_builds=1 worker=0
Feeding runners to channel                          builds=0 max_builds=1
ERROR: Failed to load config stat /home/mibo/.gitlab-runner/config.toml: no such file or directory  builds=0 max_builds=1
Feeding runners to channel                          builds=0 max_builds=1
ERROR: Failed to load config stat /home/mibo/.gitlab-runner/config.toml: no such file or directory  builds=0 max_builds=1

No jobs can be executed!
I then stoped the runner again and used

sudo gitlab-runner --debug run

and got:

Runtime platform                                    arch=amd64 os=linux pid=25611 revision=bda84871 version=18.5.0
Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0 max_builds=0
Checking runtime mode                               GOOS=linux uid=0
Running in system-mode.                            
                                                   
Usage logger disabled                               builds=0 max_builds=1
Configuration loaded                                builds=0 max_builds=1
WARNING: CONFIGURATION: Long polling issues detected.
Issues found:
  - Request bottleneck: 1 runners have request_concurrency=1, causing job delays during long polling
This can cause job delays matching your GitLab instance's long polling timeout.
Recommended solutions:
  1. Increase 'request_concurrency' to 2-4 for 1 runners currently using request_concurrency=1
Note: The 'FF_USE_ADAPTIVE_REQUEST_CONCURRENCY' feature flag can help automatically adjust request_concurrency based on workload.
This message will be printed each time the configuration is reloaded if the issues persist.
See documentation: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#long-polling-issues  builds=0 max_builds=1
listenaddress: ""
sessionserver:
  listenaddress: ""
  advertiseaddress: ""
  sessiontimeout: 1800
labels: {}
concurrent: 1
checkinterval: 0
loglevel: null
logformat: null
user: ""
runners:
- name: Ing-Buro-Server
  limit: 0
  outputlimit: 0
  requestconcurrency: 0
  unhealthyrequestslimit: 0
  unhealthyinterval: null
  jobstatusfinalupdateretrylimit: 0
  systemid: s_4f6ed7f08ac4
  configloadedat: 2025-10-23T21:05:22.988987045+02:00
  configdir: /etc/gitlab-runner
  runnercredentials:
    url: http://127.0.0.1
    id: 1
    token: '[MASKED]'
    tokenobtainedat: 2025-09-03T19:58:51Z
    tokenexpiresat: 0001-01-01T00:00:00Z
    tlscafile: ""
    tlscertfile: ""
    tlskeyfile: ""
    logger: null
  runnersettings:
    labels: {}
    executor: shell
    buildsdir: ""
    cachedir: ""
    cloneurl: ""
    environment: []
    proxyexec: null
    pregetsourcesscript: ""
    postgetsourcesscript: ""
    prebuildscript: ""
    postbuildscript: ""
    debugtracedisabled: false
    safedirectorycheckout: null
    cleangitconfig: null
    shell: ""
    custombuilddir:
      enabled: null
    referees: null
    cache:
      type: ""
      path: ""
      shared: false
      maxuploadedarchivesize: 0
      s3:
        serveraddress: ""
        accesskey: ""
        secretkey: ""
        sessiontoken: ""
        bucketname: ""
        bucketlocation: ""
        insecure: false
        authenticationtype: ""
        serversideencryption: ""
        serversideencryptionkeyid: ""
        dualstack: null
        pathstyle: null
        accelerate: false
        rolearn: ""
        uploadrolearn: ""
      gcs:
        cachegcscredentials:
          accessid: ""
          privatekey: ""
        credentialsfile: ""
        bucketname: ""
      azure:
        cacheazurecredentials:
          accountname: ""
          accountkey: ""
        containername: ""
        storagedomain: ""
    gracefulkilltimeout: null
    forcekilltimeout: null
    featureflags: {}
    monitoring: null
    instance: null
    ssh: null
    docker: null
    parallels: null
    virtualbox: null
    machine: null
    kubernetes: null
    custom: null
    autoscaler: null
    steprunnerimage: ""
sentrydsn: null
connectionmaxage: 15m0s
modtime: 2025-09-03T22:00:05.3107137+02:00
loaded: true
experimental: null
shutdowntimeout: 0
configsaver: null
  builds=0 max_builds=1
Waiting for stop signal                             builds=0 max_builds=1
listen_address not defined, metrics & debug endpoints disabled  builds=0 max_builds=1
[session_server].listen_address not defined, session endpoints disabled  builds=0 max_builds=1
Initializing executor providers                     builds=0 max_builds=1
Feeding runners to channel                          builds=0 max_builds=1
Feeding runner to channel                           builds=0 max_builds=1 runner=vHNZf35rK
Starting worker                                     builds=0 max_builds=1 worker=0
Processing runner                                   builds=0 max_builds=1 runner=vHNZf35rK
Acquiring job slot                                  builds=0 max_builds=1 runner=vHNZf35rK
Acquiring request slot                              builds=0 max_builds=1 runner=vHNZf35rK
Acquiring executor from provider                    builds=0 max_builds=1 runner=vHNZf35rK
Dialing: tcp 127.0.0.1:80 ...                      
Checking for jobs...no content                      correlation_id=01K897C89F4MK8N29FBCGQRRNS runner=vHNZf35rK status=204 No Content
Feeding runners to channel                          builds=0 max_builds=1
Feeding runner to channel                           builds=0 max_builds=1 runner=vHNZf35rK
Processing runner                                   builds=0 max_builds=1 runner=vHNZf35rK
Acquiring job slot                                  builds=0 max_builds=1 runner=vHNZf35rK
Acquiring request slot                              builds=0 max_builds=1 runner=vHNZf35rK
Acquiring executor from provider                    builds=0 max_builds=1 runner=vHNZf35rK
Checking for jobs...no content                      correlation_id=01K897CB79CDRQ0K3J6686A9PQ runner=vHNZf35rK status=204 No Content
Feeding runners to channel                          builds=0 max_builds=1
Feeding runner to channel                           builds=0 max_builds=1 runner=vHNZf35rK
Processing runner                                   builds=0 max_builds=1 runner=vHNZf35rK
Acquiring job slot                                  builds=0 max_builds=1 runner=vHNZf35rK
Acquiring request slot                              builds=0 max_builds=1 runner=vHNZf35rK
Acquiring executor from provider                    builds=0 max_builds=1 runner=vHNZf35rK

when i execute a job i got the following from debug-runner log:

Acquiring job slot                                  builds=0 max_builds=1 runner=vHNZf35rK
Acquiring request slot                              builds=0 max_builds=1 runner=vHNZf35rK
Acquiring executor from provider                    builds=0 max_builds=1 runner=vHNZf35rK
Checking for jobs... received                       correlation_id=01K897HXFMKY44HSY207D6STP5 job=48 repo_url=http://127.0.0.1/ingburo/ifen/applications/stx-fpga.git runner=vHNZf35rK
Updating job...                                     bytesize=0 checksum= job=48 runner=vHNZf35rK
Submitting job to coordinator...ok                  bytesize=0 checksum= code=200 correlation_id=01K897HXNTS17X0P607V7RP75S job=48 job-status=running runner=vHNZf35rK update-interval=0s
Added job to processing list                        builds=1 gitlab_user_id=3 job=48 max_builds=1 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga queue_depth=1 queue_size=1 repo_url=http://127.0.0.1/ingburo/ifen/applications/stx-fpga.git root_namespace_id=5 runner=Ing-Buro-Server time_in_queue_seconds=1
Failed to requeue the runner                        builds=1 max_builds=1 runner=vHNZf35rK
Running with gitlab-runner 18.5.0 (bda84871)        gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
  on Ing-Buro-Server vHNZf35rK, system ID: s_4f6ed7f08ac4  gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Preparing the "shell" executor          gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Shell configuration: command: bash
arguments:
- -l
cmdline: bash -l
dockercommand:
- sh
- -c
- "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash -l\nelif [ -x /usr/bin/bash
  ]; then\n\texec /usr/bin/bash -l\nelif [ -x /bin/bash ]; then\n\texec /bin/bash
  -l\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh -l\nelif [ -x
  /usr/bin/sh ]; then\n\texec /usr/bin/sh -l\nelif [ -x /bin/sh ]; then\n\texec /bin/sh
  -l\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh -l\nelse\n\techo shell not
  found\n\texit 1\nfi\n\n"
passfile: false
extension: ""
  gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Using Shell (bash) executor...                      gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Waiting for signals...                              gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
No referees configured                              gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Executing build stage                               build_stage=prepare_script gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Preparing environment                   gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Using new shell command execution                   gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Executing build stage                               build_stage=get_sources gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Getting source from Git repository      gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Using new shell command execution                   gitlab_user_id=3 job=48 namespace_id=13 organization_id=1 project=15 project_full_path=ingburo/ifen/applications/stx-fpga root_namespace_id=5 runner=vHNZf35rK
Feeding runners to channel                          builds=1 max_builds=1
Feeding runner to channel                           builds=1 max_builds=1 runner=vHNZf35rK
Appending trace to coordinator...ok                 code=202 correlation_id=01K897J0KK5C9W5RRYAKH3S9Q7 job=48 job-log=0-32170 job-status=running runner=vHNZf35rK sent-log=0-32169 status=202 Accepted update-interval=3s
Updating job...                                     bytesize=32170 checksum=crc32:1184bebd job=48 runner=vHNZf35rK
Submitting job to coordinator...ok                  bytesize=32170 checksum=crc32:1184bebd code=200 correlation_id=01K897JXZNSSYMVKS10B6YRWKD job=48 job-status=running runner=vHNZf35rK update-interval=0s

and the buildjob seems to stuck when i look at the pipeline output (same also after 30min.):

Running with gitlab-runner 18.5.0 (bda84871)
  on Ing-Buro-Server vHNZf35rK, system ID: s_4f6ed7f08ac4
Preparing the "shell" executor 00:00
Using Shell (bash) executor...
Preparing environment 00:00
....
....
...
++ '[' -d /home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga ']'
++ chmod -R u+rwX /home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga
++ rm -r -f /home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga
Fetching changes with git depth set to 20...
++ echo 'Fetching changes with git depth set to 20...'
++ git init /home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga --template /home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga.tmp/git-template
Initialized empty Git repository in /home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga/.git/
++ cd /home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga
++ git remote add origin http://gitlab-ci-token:[MASKED]@127.0.0.1/ingburo/ifen/applications/stx-fpga.git
Created fresh repository.
++ echo 'Created fresh repository.'
++ git -c 'http.userAgent=gitlab-runner 18.5.0 linux/amd64' fetch origin --no-recurse-submodules +refs/pipelines/21:refs/pipelines/21 +refs/heads/main:refs/remotes/origin/main --depth 20 --prune --quiet
Checking out a135c96a as detached HEAD (ref is main)...
++ echo 'Checking out a135c96a as detached HEAD (ref is main)...'
++ git -c submodule.recurse=false checkout -f -q a135c96a068f3e4c07a68fea06f30449ddbe879d
++ git clean -ffdx
++ git lfs version
Updating/initializing submodules recursively with git depth set to 20...
++ echo 'Updating/initializing submodules recursively with git depth set to 20...'
++ git submodule init
Submodule 'src/vhdl-library' (git@192.168.178.122:ingburo/ifen/libraries/vhdl-library.git) registered for path 'src/vhdl-library'
++ git submodule sync --recursive
Synchronizing submodule url for 'src/vhdl-library'
++ git submodule foreach --recursive 'git clean -ffdx'
++ git submodule foreach --recursive 'git reset --hard'
++ git -c url.http://gitlab-ci-token:[MASKED]@127.0.0.1.insteadOf=http://127.0.0.1 -c url.http://gitlab-ci-token:[MASKED]@127.0.0.1/.insteadOf=git@127.0.0.1: -c url.http://gitlab-ci-token:[MASKED]@127.0.0.1.insteadOf=ssh://git@127.0.0.1 submodule update --init --recursive --depth 20
Cloning into '/home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga/src/vhdl-library'...

when i do then

gitlab-runner stop
gitlab-runner restart

everything is as before.

9

Thanks for the details, interesting to see the git command with .insteadof


++ git submodule init
Submodule 'src/vhdl-library' (git@192.168.178.122:ingburo/ifen/libraries/vhdl-library.git) registered for path 'src/vhdl-library'
++ git submodule sync --recursive
Synchronizing submodule url for 'src/vhdl-library'
++ git submodule foreach --recursive 'git clean -ffdx'
++ git submodule foreach --recursive 'git reset --hard'
++ git -c url.http://gitlab-ci-token:[MASKED]@127.0.0.1.insteadOf=http://127.0.0.1 -c url.http://gitlab-ci-token:[MASKED]@127.0.0.1/.insteadOf=git@127.0.0.1: -c url.http://gitlab-ci-token:[MASKED]@127.0.0.1.insteadOf=ssh://git@127.0.0.1 submodule update --init --recursive --depth 20
Cloning into '/home/mibo/builds/vHNZf35rK/0/ingburo/ifen/applications/stx-fpga/src/vhdl-library'...

I don’t know what’s wrong. Might need a reproducible project but I am pretty overloaded with work tasks right now, can take some time.

Can you reproduce your problem in 2 simple projects on gitlab.com, one main, one referenced as ssh submodule? Does not have to be VHDL code nor Xilinx CI config.

10

I have not used VHDL since my studies in 2005, so this was a fun project to let Duo Agentic Chat generate example code to reproduce the problem.

Create submodule dependency: VHDL library

image
image1876×1293 274 KB

image
image1876×1293 327 KB

Result: Developer Advocacy at GitLab / playground / dnsmichi / Submodule Test - Dep - VHDL library · GitLab it comes with its own README and CI/CD so could serve for other use cases as well.

Use dependency in main VHDL builder project

Use the VHDL library as submodule

In the main project, I’ve run the following command to add the git submodule

git submodule add git@gitlab.com:gitlab-da/playground/dnsmichi/submodule-test-dep-vhdl-library.git src/vhdl-library

and then asked Duo Agentic Chat to help me build it in CI/CD.

image
image1869×1208 396 KB

That creates the foundation to test the submodule override further. Developer Advocacy at GitLab / playground / dnsmichi / Submodule Test - Main - VHDL Builder · GitLab

Submodules with SSH in CI/CD

As expected, I ran into submodule setup problems with SSH only, and needed to configure GIT_SUBMODULE_FORCE_HTTPS - you can see the commit history in Add VHDL library submodule, and build CI/CD in main project (!1) · Merge requests · Developer Advocacy at GitLab / playground / dnsmichi / Submodule Test - Main - VHDL Builder · GitLab where this is fixed amongst other things with the source code and CI/CD config.

main branch is stable and CI/CD green OK.

image
image910×398 59.4 KB

Use SSH git submodules - disable GIT_SUBMODULE_FORCE_HTTPS

Now that the repository works and CI/CD OK, adding another test with disabling the variable again to let SSH clones happen for submodules - Disable GIT_SUBMODULE_FORCE_HTTPS (!2) · Merge requests · Developer Advocacy at GitLab / playground / dnsmichi / Submodule Test - Main - VHDL Builder · GitLab As expected, it fails.

image
image816×411 64.4 KB

Why does it work in my GitLab.com environment?

  1. Server: GitLab.com rolling release (should not have an influence here)
  2. Runner: Running with gitlab-runner 18.4.0~pre.195.g5de42c65 (5de42c65) on blue-1.saas-linux-small.runners-manager.gitlab.com/default j1aLDqxSn, system ID: s_ccdc2f364be8

.gitlab-ci.yml

variables:
  # Submodule configuration
  GIT_SUBMODULE_STRATEGY: recursive
  GIT_SUBMODULE_UPDATE_FLAGS: --remote
  GIT_SUBMODULE_FORCE_HTTPS: "true"

Next steps:

  1. To rule out a GitLab Runner version bug with 18.5, I’ve used my Ansible environment to install and register a new GitLab Runner on a VM.

Runner 18.5 on Ubuntu

Installation based on ansible-gitlab-gitlab-runner-raspberry-pi which uses the same OS (Debian). Hacky workaround but faster than manual setup. michi.fyi is an existing Ubuntu VM with base OS and Docker. You can see the changes in this draft MR, in case that is helpful and not too overwhelming. I prefer to be as transparent as possible, except for secrets in Ansible Vault.

Registered runner

Settings > CI/CD > Runners

image
image753×427 36.6 KB

image
image437×166 12.2 KB

Test CI/CD with new 18.5 runner

To force all jobs on the project runner, it was registered with the ubuntu tag, and all CI/CD jobs use this tag. See this MR CI: Run all jobs on Project runner ubuntu (!3) · Merge requests · Developer Advocacy at GitLab / playground / dnsmichi / Submodule Test - Main - VHDL Builder · GitLab - code_quality

Shell executor

MR with CI_DEBUG_TRACE enabled

code_quality job

echo $'\''\x1b[32;1mUpdating/initializing submodules recursively with git depth set to 20...\x1b[0;m'\''
git submodule init
git submodule sync --recursive
git submodule foreach --recursive $'\''git clean -ffdx'\''
git submodule foreach --recursive $'\''git reset --hard'\''
if git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule update --init --recursive --depth 20 --remote; then
  echo $'\''\x1b[32;1mUpdated submodules\x1b[0;m'\''
  git submodule sync --recursive
else
  echo $'\''\x1b[0;33mUpdating submodules failed. Retrying...\x1b[0;m'\''
  git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule foreach --recursive $'\''git fetch origin +refs/heads/*:refs/remotes/origin/*'\''
  git submodule sync --recursive
  git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule update --init --recursive --depth 20 --remote
  git submodule foreach --recursive $'\''git reset --hard'\''
fi
git submodule foreach --recursive $'\''git clean -ffdx'\''
if git lfs version >/dev/null 2>&1; then
  git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule foreach --recursive $'\''git lfs pull'\''
fi
if [ -e "/builds/.gitlab-build-uid-gid" ]; then
  if [ -d "/builds" ]; then
    chown -R "$(stat -c '\''%u:%g'\'' '\''/builds/.gitlab-build-uid-gid'\'')" '\''/builds'\''
    echo "Setting ownership for /builds to $(stat -c '\''%u:%g'\'' '\''/builds/.gitlab-build-uid-gid'\'')"
  fi
fi

image
image900×1044 164 KB

Docker executor

MR with CI_DEBUG_TRACE enabled

code_quality job:

echo $'\''\x1b[32;1mUpdating/initializing submodules recursively with git depth set to 20...\x1b[0;m'\''
git submodule init
git submodule sync --recursive
git submodule foreach --recursive $'\''git clean -ffdx'\''
git submodule foreach --recursive $'\''git reset --hard'\''
if git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule update --init --recursive --depth 20 --remote; then
  echo $'\''\x1b[32;1mUpdated submodules\x1b[0;m'\''
  git submodule sync --recursive
else
  echo $'\''\x1b[0;33mUpdating submodules failed. Retrying...\x1b[0;m'\''
  git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule foreach --recursive $'\''git fetch origin +refs/heads/*:refs/remotes/origin/*'\''
  git submodule sync --recursive
  git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule update --init --recursive --depth 20 --remote
  git submodule foreach --recursive $'\''git reset --hard'\''
fi
git submodule foreach --recursive $'\''git clean -ffdx'\''
if git lfs version >/dev/null 2>&1; then
  git -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=https://gitlab.com'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com/.insteadOf=git@gitlab.com:'\'' -c $'\''url.https://gitlab-ci-token:[MASKED]@gitlab.com.insteadOf=ssh://git@gitlab.com'\'' submodule foreach --recursive $'\''git lfs pull'\''
fi
if [ -e "/builds/.gitlab-build-uid-gid" ]; then
  if [ -d "/builds" ]; then
    chown -R "$(stat -c '\''%u:%g'\'' '\''/builds/.gitlab-build-uid-gid'\'')" '\''/builds'\''
    echo "Setting ownership for /builds to $(stat -c '\''%u:%g'\'' '\''/builds/.gitlab-build-uid-gid'\'')"
  fi
fi

image
image900×1044 164 KB

Runner config

No special GitLab Runner configuration except for using Docker and Shell as executor, attaching it here so you can compare.

michi@michi:~$ sudo cat /etc/gitlab-runner/config.toml

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "michi.fyi"
  url = "https://gitlab.com"
  id = 50285641
  token = "glrt-xxxxx"
  token_obtained_at = 2025-10-25T14:26:31Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0

[[runners]]
  name = "michi.fyi"
  url = "https://gitlab.com"
  id = 50285697
  token = "glrt-xxxxxxx"
  token_obtained_at = 2025-10-25T14:49:29Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Test results

SSH submodules with HTTPS override work in CI/CD in different scenarios.

  1. GitLab Runner 18.4 on GitLab.com shared runners :white_check_mark:
  2. GitLab Runner 18.5 on bare Ubuntu VM

Conclusion

I suspect a host environment problem with git, not a bug. Sharing a few ideas to debug further:

Next steps to debug

  1. Test with a Docker executor, too, and run jobs in containers.
  2. Test with a fresh host OS + Runner and shell executor
  3. Host OS:
  • Security policies interfering, like SELinux or AppArmor (just thinking out loud).
  • When the shell executor is used, maybe the host OS interferes with Git settings that override the Git commands executed inside the runner? Check /etc/gitconfig and ~/.gitconfig
  • Git installed on the host OS too old/buggy. Run git --version

My personal recommendation: When possible, isolate jobs into containers using the docker executor, and pass artifacts with dependencies into jobs that require host OS access (firmware flash to connected USB/Serial devices, etc.), they can use the shell executor. I’ve shared a similar example in my Container Days talk recently, slides in

11

Hi Michael,

wow - thanks for the very detailed analysis. I will try what you suggest during the next days.
I am very busy with other projects at the moment but will try my best.

I will let you know the outcome as soon as possible.

Test with a fresh host OS + Runner and shell executor
→ That’s what i already done. Everything is fresh installed incl. OS. I also use the shell executor.

Thanks a lot, Michael