Repository mirroring, two Gitlab CE instances

Hi there,

we are trying to mirror git projects between two Gitlab CE instances. Requirements:

  • push mirroring between the instances
  • no bi-directional mirroring necessary, target instance is read-only
  • we want to mirror ALL projects in <source gitlab> (except possibly private projects) to a group namespace in <target gitlab>

Our approach:

  • Set up nightly cron job which uses the Gitlab API to create a repository in <target gitlab> if it doesn’t exist and sets up corresponding repository push mirroring in <source gitlab>

Our problem:

  • How can we handle frequent changes of project names and name spaces in <source gitlab>?

Since IDs uniquely identify projects, an idea is to mirror repositories to <target gitlab>/<group>/<project_id>/<namespace>/<project_name>. However, then a change of project name or name space would lead to a change in mirror URL and lead to orphaned projects.

Is there a more elegant way to do this?

Hi,

I would build in index/map for IDs=>names before updating the target. That way you can detect a name change, but same ID, and update the projects accordingly then.

Either by getting a single project by ID from both the source and target, or with a whole list and building the index.

If you detect such a naming change then, you probably need to decide whether to keep using the old name, or update it in a secondary API call route, next to the first with setting up the sync.

Might not be the most elegant way as it involves some scripting, but I don’t see another way of keeping this in sync here.

Cheers,
Michael

1 Like

How about using a ci job to push the changes on demand?

1 Like

Thanks for your input! To clarify, we don’t care about IDs in the target Gitlab. The list would include source ID and target namespace + project name. We could then compare namespace + project name in the source to namespace + project name in the target. If they don’t match -> create new project in target Gitlab and update repository mirror in source Gitlab accordingly.

1 Like

Thank you! You mean include CI jobs with each project we want to mirror? That’s not really feasible, we have more than 100 users who create and maintain their own projects. If you mean a central CI job - how would we be able to detect repository changes and trigger pushes?

For an alternative please see https://docs.gitlab.com/ee/administration/server_hooks.html#set-a-global-server-hook-for-all-repositories

This will then not be visible in the ci, but can be performed in parallel

2 Likes

Excellent addition, @kaystrobach!

@rfreund Let us know if you are closer to a solution! We’re still here if you need more help troubleshooting! :blush: