Git clone of other repos during CI/CD process

Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

During our PREBUILD step - we need to use GIT to pull other projects from the GITLAB server.

Steps to reproduce

Example in a PREBUILD script step - I need to “git clone” other repos on the same GIT server the runner is fetching from.

My belief is this: the GITLAB RUNNER can PULL the target repo across the runner interface, how do I “convert/re-encode” a GIT URL to use the same interface/transport as the runner? Which, I presume is not SSH and NOT HTTP - but something else.

ie: I normally would “git clone git@gitlab.mycompany.local:somegroup/project.git”

that works if and only if the RUNNER has a KEY pre installed.

Instead, the runner has already authenticated, and can pull the current project. How can use the SAME authentication tunnel to pull other things from the same GIT LAB SERVER?

NOTE: our operation requires that we only use SSH for GIT access, with the exception the GITLAB RUNNER uses its own internal authentication with an ACCESS TOKEN.

Stated differently:
Git lab RUNNER has used the access token to PULL the [primary] project from the server.
HOW - can I make GIT use the same “access token” to pull other projects on the same server across the access/token/tunnel?

I understand that if I am talking to a DIFFERENT git server I would need authentication for that server - but that is not the case here. I am requesting a GIT repo over the same INTERFACE to the SAME server.

Thanks.

Hi,

Firstly, there is no “special” clone between GitLab Runner and GitLab. It’s a simple clone over HTTPS using short-lived CI_JOB_TOKEN (even if https is disabled on instance level)

I never tried this using the CI_JOB_TOKEN, but it might be possible if you go to every project’s (that you need to clone) Settings > CI/CD > Job token permissions and add the project where the pipeline is being executed to the Allowlist.

Then in the pipeline’s job, you will manually add other projects. E.g.

my-job:
  script:
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>
    - # the rest of the script

If you’re using git sub-modules, you can tell GitLab to follow them:

variables:
  GIT_SUBMODULE_STRATEGY: recursive

To cache repositories between the prebuild job and other jobs in GitLab CI/CD, you can use GitLab’s built-in caching mechanism. Here’s how you can set it up:

1. Define a cache in your .gitlab-ci.yml file:

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - cached-repos/

2. In your prebuild job, clone the repositories into the cached-repos/ directory:

prebuild:
  stage: .pre
  script:
    - mkdir -p cached-repos
    - cd cached-repos
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/group/repo1.git
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/group/repo2.git
  cache:
    policy: push

3. In subsequent jobs, you can use the cached repositories:

build:
  stage: build
  script:
    - cd cached-repos/repo1
    - # Use repo1
    - cd ../repo2
    - # Use repo2
  cache:
    policy: pull

This requires that git is available in your build images. This will:

• Cache the cloned repositories in the cached-repos/ directory.
• Push the cache in the prebuild job.
• Pull the cache in subsequent jobs, making the cloned repositories available.

1- In order to clone a repo directly you can’t do that, because the runner machine isn’t logged in into gitlab.com or it doesn’t have ssh authentication.

2- you can do it using ${CI_JOB_TOKEN}, and to be able to clone the repo using ${CI_JOB_TOKEN}, the person who initiate the pipeline’s run (commits, merge, run manually) he need to have a higher role, he needs to have the rights to be able to clone

3- I suggest to do it with deploy_tokens :
- go to the repo you want to clone → Settings → Repository → Deploy Tokens → Add token
- give it a name and check only ‘read_repository’
- copy the deploy_username and deploy_token that were generated
- git clone --branch “branch-name” “https://${deploy_username}:${deploy_token}@gitlab.com/path_to_project”