Backup restore procedure test before upgrade 🚀


I’m trying to test some backup/restore procedures before I upgrade our Omnibus install from 13.2.1 to the latest 13.x version. But when I restore a backup (with verified content) to a blank instance, nothing is being restored.

We’re using the omnibus install 13.2.1 with an external Postgres container (v12). This setup is deployed on a single node docker swarm as a docker-compose stack.

Knowing that we’re using a external Postgres I followed this article → backup-and-restore-a-non-packaged-postgresql-database, but this failed because we’re using a higher Postgres version (v12) in our database container (whoops…). The Omnibus install has Postgres 11.7. I tied to copy over the psql and pg_dump executables to the gitlab container but got all sorts of library errors (missing and wrong versions). So that’s why I skiped the DB backup in the gitlab-rake command and perform a seperate Postgres dump on the Postgres container. Hope that this is ok… :slight_smile:

The backup was created with the following command’s:

docker exec -i <gitlab_container_id> gitlab-ctl reconfigure # -> (to process any left-over's in the config)
docker exec -i <gitlab_container_id> gitlab-rake gitlab:backup:create SKIP=db --trace

docker cp <gitlab_container_id>:/var/opt/gitlab/backups/<backup_file>.tar ./

docker cp <gitlab_container_id>:/etc/gitlab/gitlab.rb ./
docker cp <gitlab_container_id>:/etc/gitlab/gitlab-secrets.json ./
docker cp <gitlab_container_id>:/etc/omnibus_config.rb ./

docker exec -t <postgres_container_id> pg_dumpall -c -U gitlab | gzip > database_dump.gz

And I tried to restore with:

docker cp ./gitlab.rb <gitlab_container_id>:/etc/gitlab/gitlab.rb
docker cp ./gitlab-secrets.json <gitlab_container_id>:/etc/gitlab/gitlab-secrets.json
docker cp ./omnibus_config.rb <gitlab_container_id>:/etc/omnibus_config.rb    # -> Gives a file in use error...

docker exec -i <gitlab_container_id> gitlab-ctl reconfigure
docker exec -i <gitlab_container_id> gitlab-ctl stop nginx
docker exec -i <gitlab_container_id> gitlab-ctl stop sidekiq

BACKUP_TAG=${BACKUP_FILE/_gitlab_backup.tar/} # -> remove _gitlab_backup.tar from filename

docker exec -i <gitlab_container_id> gitlab-rake gitlab:backup:restore BACKUP=${BACKUP_TAG} SKIP=db force=yes --trace

gunzip < database_dump.gz | docker exec -i <postgres_container_id> psql -U gitlab -d gitlab

docker exec -i <gitlab_container_id> gitlab-ctl restart
docker exec -i <gitlab_container_id> gitlab-ctl reconfigure
docker exec -i <gitlab_container_id> gitlab-rake gitlab:check SANITIZE=true

# Check if web service is running:
docker service logs -f gitlab_web

After the restore I hoped to find my “Test group”/“Test project” with some pre-populated scripts/content in the web-ui. But nothing is there. The restore command doesn’t list the group/repo as being restored, it did get listed at the backup stage.

To verify the content of the backup I extracted the .tar file and git cloned the <random_name>.bundle to a folder. The test scripts/content were visible after the clone. So the backup should be good.

Am I missing something?