Automated Copy of One Repo Branch to Another Repo Branch without Forking

We need some help automating the following.

We have two repositories

  • Code: Contains all actual code under development
  • Reports: Contains one-off SQL code scripts for reporting based on requests from customers

Within the Code repository contains SQL template files that in turn get used by the people working in the Reports repository. When a report writer gets a request they create a branch in the Reports repository for that request, select a template, fill in the blanks then generate and deliver the report.

What we want to do is automatically copy the main branch of the Code repository over to the Reports repository so that every time a report writer gets a new request they start with the latest approved version of the templates.

We started with an automated mirror/fork of Code/main to Reports/main but that has caused a lot of problems. The biggest problem is because Reports is a fork of Code merge requests from branches in Reports by default target Code/main. We could probably train our report writers on how to avoid this but it would always be subject to human error.

What we would love to do automatically copy the contents of (with or without history) of Code/main to Reports/main in such a way that nothing that occurs over in the Reports repository would accidentally get pushed over into Code/main.

A boring solution can be using Git CLI commands, and wrap these into shell scripts, which are called by a cronjob (Linux VM or CI/CD schedules).

Requirements

  1. Script can fetch the Code project
  2. Script can write the Reports project
  3. Git CLI auth (local with SSH or HTTPs as transport)

Important: Use an account whose role is allowed to force push protected branches in the Reports project.

Script

I did not test the script, but it should work in a cronjob. Please test it with some test projects first.

#!/bin/bash

mkdir working-dir
cd working-dir

# Do a fresh clone to fetch the latest main branch automatically 
git clone https://gitlab.server.com:namespace/Code.git

cd Code

git checkout main

git remote add target https://gitlab.server.com:namespace/Reports.git
git fetch target 

# Push the local Code main branch into the target remote, same branch name `main` 
# Force push to override 
git push -f main:target/main 

cd ../..

rm -rf working-dir 

Alternatively, you can trigger a CI/CD pipeline which executes the script, when someone pushes Git commits into the main branch of the Code project. Note that the pipeline Git user will need the same authorization setup noted above.

Some more scripts are discussed in Refreshing a Fork - #19 by dnsmichi

Interesting. @dnsmichi is there any way to automate this entirely on gitlab.com? Would that be the CI/CD pipeline route you mentioned?

Yes. The idea is to use CI/CD schedules, which trigger a CI/CD pipeline and jobs. These jobs execute a script section with commands - i.e. setting up the Git CLI, running git fetch, and then git push. The commands are executed on the GitLab.com SaaS shared runners (will affect your minutes of compute usage).

Something like this in a separate project and .gitlab-ci.yml

sync-main-to-reports:
  script:
    - # TODO: git setup steps here. Either ssh key or PAT, via CI/CD variables. 
    - git clone https://gitlab.server.com:namespace/Code.git && cd Code
    - git remote add target https://gitlab.server.com:namespace/Reports.git
    - git fetch target 
    - git push -f main:target/main  
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

The job is only run when triggered via pipeline schedule (see Choose when to run jobs | GitLab)

For SSH keys setup in CI/CD, please follow Using SSH keys with GitLab CI/CD | GitLab