So, I have managed to recover my system using restic. The backup/snapshot was taken whilst Gitlab was running. This does mean some caveats to getting it running again, but it is doable. Browsing the web interface, I do see my repositories, and I do see the files listed in them, and I can view them.
My process for the backup was:
restic backup /etc/gitlab /var/opt/gitlab
This is then shown in the output below:
root@gitlab:~# restic snapshots
repository e07706e0 opened (version 2, compression level auto)
ID Time Host Tags Paths
----------------------------------------------------------------------
e074be8d 2024-01-24 18:57:19 gitlab /etc/gitlab
/var/opt/gitlab
----------------------------------------------------------------------
1 snapshots
I then rsynced this across to my new server. The snapshots I stored in /var/restic but you can choose the location yourself when you use the restic init command. Once on the new server:
root@debian:~# restic snapshots
repository e07706e0 opened (version 2, compression level auto)
ID Time Host Tags Paths
----------------------------------------------------------------------
e074be8d 2024-01-24 18:57:19 gitlab /etc/gitlab
/var/opt/gitlab
----------------------------------------------------------------------
1 snapshots
My first step was to install gitlab-ce-16.8.0 which is what I was running on the old server. Once this was installed, I then ran the restic restore procedure:
root@debian:/var/restic# restic restore e074be8d --target /
repository e07706e0 opened (version 2, compression level auto)
[0:00] 100.00% 1 / 1 index files loaded
restoring <Snapshot e074be8d of [/etc/gitlab /var/opt/gitlab] at 2024-01-24 18:57:19.557248311 +0100 CET by root@gitlab> to /
Summary: Restored 10810 files/dirs (1.153 GiB) in 0:02
At this point, I was able to run gitlab-ctl reconfigure and get my Gitlab install sane enough in terms of configuration. Some pid files needed to be removed from the postgresql and gitaly directories because they existed in the backup because of it being online.
After this you will be debugging a few things in terms of permissions, for example, access to git-data, postgresql, redis. I did use the gitlab-rake gitlab:check command to also ensure everything had restored - this also hinted and fixing authorized_keys and one or two other things as well.
Depending on which service is failing to start, you can check the log file output and it will usually be throwing permission errors. I had that for postgres, gitaly, redis and some issues with nginx since I changed the hostname of the server, but not the names of the certificate files.
Also, depending on whether you can afford downtime overnight, you could script it to stop Gitlab before running the restic backup. That would at least help with the PID file issue. Then once the backup/snapshot has finished start Gitlab again. I decided on the harder route, just to prove it’s possible to be done as online backup - at least in my tests but your mileage may vary.
Version of restic I used was not the one that is installable from distro repositories, but went to their downloads and got 0.16.3 which is the latest as of this post.