Need help to recover my Gitlab CE instance via only a filesystem backup

Intro - System failure and data loss

A few days ago, I had multiple disks in my system to stop functioning. My system crashed and shut down. I luckily have good backup routines (or so I thought), until I discovered that it is not very easy to get GitLab with omnibus, back up running just by having the files backed up. I yesterday read a bit how backups should be done, and that there is a tool for creating backups for gitlab, as well as Prometheus backups(?).

What do I currently have?

  • /etc/gitlab/*
  • /var/opt/gitlab/*
  • + a bunch of other directories, but these are related to gitlab if I am not mistaken.

I am currently trying to recover my instance to a Ubuntu Server 20.04 setup.

What did I try so far?

I have attempted to first make sure that I install the exact same version of GitLab that I had running when the accident happened. That was GitLab CE 13.12.3 released June 7th. After, I attempted to copy my contents of /etc and /var to the respective folders, and try getting GitLab to understand to use these files :slight_smile: But - against my luck, that did not seem to work. I only get 502’s, and it looks like prometheus doesn’t want to start.

Question

I am wondering if someone in here with a little better understanding of Prometheus and GitLab can help me find a way to recover my work?

In summary I am quite frightened that I will have to learn the hard way by losing many of my dear and near projects I kept storing locally, but I still hope there is a chance for me to recover it.

I am supper appreciative of any assistance, and good tip I can get on my journey to restore everything. I know the files are there, they just aren’t “working” :sleepy:

1 Like

I got it solved!

So what I ended up doing to test more around was:

  1. I created a Docker-based setup using docker compose. I used this guide
  2. I installed the exact version of gitlab I used before. You can see the top of my docker-compose.yml below:
version: "2.6"
services:
  web:
    image: 'gitlab/gitlab-ce:13.12.3-ce.0'
    restart: always
    volumes:
     - './data/etc/gitlab:/etc/gitlab'
     - './data/var/log/gitlab:/var/log/gitlab'
     - './data/var/opt/gitlab:/var/opt/gitlab'

#[... + a lot of config] 
  1. I configured and started the docker container, then I made sure GitLab was fully operational and up&running before I contiued to step 4.
  2. Then I stopped the docker container.
  3. Copied all files of my old /var/opt/gitlab to the docker mount I used, replacing the existing files.
  4. I did the same for /etc/gitlab/gitlab-secrets.json
  5. Started the container again, waited 5 minutes to let it start up properly. Proceeded to notice that my log was really full of permission errors.
  6. Identified my docker container name with $ docker ps. It was “gitlab_web_1”.
  7. I updated all my file permissions with docker exec -it gitlab_web_1 update-permissions
  8. Some files didn’t bite, those i sat manually with: docker exec -it gitlab_web_1 chown -R git:git /var/opt/gitlab/gitaly/, as they were in that folder.
  9. I restarted my GitLab container. and everything worked out!

I documented my process here in case it can be helpful for someone later. I think it should be fairly similar directly using omnibus, but what I didn’t know was how to set the correct file permissions on the files.

Because of my success, I will continue to use a docker based gitlab setup from now on.

den

3 Likes