Gitlab-rake gitlab:backup:create VS Gitlab-backup

I’m running Gitlab v15.7.5 on a CentOS7 VM with 4cpus and 4Go of RAM
Every night we run a cron to make a backup of our Gitlab instance.

The command we use is the following :

gitlab-rake gitlab:backup:create SKIP=registry,artifacts CRON=1

We use an ELK stack to monitor our VM and by using this command we observe a light spike at 1:00 AM which is the hour where the cron is triggered

This command is fine for us, but we are aware that this one is deprecated like mentionned in this article : Back up GitLab | GitLab (Version 12.1 and earlier)

So we tried to switch on the new recommanded backup command for version 12.2 and after :

sudo gitlab-backup create

Unfortunately this command make our VM crash, and go full in every ressource (RAM,cpu,workload,SWAP) => This cause Gitlab to crash and the VM too :

I tried to add the following parameter : GITLAB_BACKUP_MAX_CONCURRENCY=1 but it didn’t work.

We went back to our previous backup process, but regarding the EOL of the previous command we have to find a way to make the new one successful. We have 2 questions :

  • When will the old command (gitlab-rake backup ) will be disabled ?
  • Is there anyone else who observed this behaviour with the new backup command ? What can I try to make it better ?

Thanks !

There should be little or no difference in behavior, as gitlab-backup create is just a wrapper around gitlab-rake gitlab:backup:create (omnibus-gitlab:config/templates/gitlab-rails/rake_backup_wrapper.erb) :thinking:

Do you see the same behavor when running the gitlab-backup create command with the same options you used with gitlab-rake gitlab:backup:create?

sudo gitlab-backup create SKIP=registry,artifacts CRON=1

Thank you for your answer greg, we tried again with

sudo gitlab-backup create SKIP=registry,artifacts CRON=1

Unfortunately the result is the same, we just restored the VM and we are ready to investigate again. Any suggestion or hint is welcome from now on :slight_smile:

I got some new hints to work with.
I’m going to try and play with those parameters in the gitlab.rb and update this post with my investigations

gitlab.rb
===================================
puma['per_worker_max_memory_mb'] = 2048
gitlab_rails['max_request_duration_seconds'] = 3600
puma['worker_timeout'] = 3601
puma['worker_processes'] = 8
gitlab_rails['env'] =
{    'GITLAB_RAILS_RACK_TIMEOUT' => "3600",  }

We found the solution to our problem
We replaced the rake command with the gitlab-backup command in our cron backup script, but the gitlab-backup command call the gitlab-backup script in /bin/
Problem is, this script is a symbolic link of our cron backup script in our environment, thus, made it a loop…
Once the gitlab-backup script restored to the default one, and our cron-backup script moved somewhere else, everything worked as intended