How to effectively take incremental/delta backups of GitLab CE repositories?

Community GitLab for Open Source
1

Hi GitLab Community!

I’m using GitLab CE (self-hosted, version 17.6), and I’d like to ensure my repository backups are incremental (only the changes, not full backups each time).

I see that:

  • Incremental repository backups are supported (INCREMENTAL=yes) starting from GitLab 15.3.
  • However, even with INCREMENTAL=yes and a valid PREVIOUS_BACKUP=<backup-id>, the backup size for the repositories (repositories.tar) seems almost the same as a full backup.
  • My usage is primarily for large Git repos with some binary files.

My questions:

  1. Is this expected behavior for GitLab incremental repo backups?
  2. Is there a better/supported way to take true delta/incremental backups of repositories in GitLab CE?
  3. Are external tools (like rsync, borg, or restic) recommended for this instead of GitLab’s built-in backup?

Any guidance or best practices would be appreciated! Thanks in advance

2

I don’t tend to use Gitlab’s functionality for incremental, differential. I do use it for full backups at least at the minute.

However, on my test server in my lab which I’ve a copy of my production server, I do use restic here. My script though, does actually stop Gitlab before running the backup commands with restic. You can do it whilst leaving Gitlab running, although how sane the backups will be I’ve no idea. Generally at night nobody is using my Gitlab instance anyway, but that said, restoring would still require cleaning up the PID files before being able to start Gitlab.

You can see a little write-up I did on this already on the forum: 16.8.0 Backup/Restore Issues - how to repair? - #14 by iwalker

My backup script looks pretty much like this:

#!/bin/bash
#
# Backup Gitlab using Restic

# Variables
RESTIC=/usr/local/bin/restic
SYSTEMCTL=/usr/bin/systemctl
BACKUP_PARMS=~/.restic-parms
BACKUP_DIRS="/etc/gitlab /var/opt/gitlab /var/log/gitlab"
BACKUP_DAYS=30

# Backup
${SYSTEMCTL} stop gitlab-runsvdir
source ${BACKUP_PARMS}
${RESTIC} backup ${BACKUP_DIRS}
${RESTIC} forget --keep-within ${BACKUP_DAYS}d --prune
${SYSTEMCTL} start gitlab-runsvdir

It will basically create backups to a max of 30 days, and then automatically forget/prune them. I would say it’s probably the better option for you if wanting to do incremental/differential, since restic will know what it’s already got in the backups made previously. Gitlab’s backup process I don’t see how it could work since it would need to compare against something. Which is most likely why you see it doesn’t really work.