[RESOLVED] Gitlab-ce Omnibus 16.10 backup failed

How to Use GitLab
1

:hugs: Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

I would take simply a full backup before make an upgrade… thing that I make since many years without issue. This morning I got an error where the backup is failing with few explaination.

Steps to reproduce

just launch:

sudo gitlab-backup create
or
sudo gitlab-backup create --trace to be more verbose

Configuration

My space disk is OK, I make a reboot, no users connected in the app. Everything should be fine.

Versions

Version 16.10 Omnibus installed on Debian 11

This is the output

2024-07-12 12:51:34 UTC – Dumping repositories … done
2024-07-12 12:51:34 UTC – Dumping uploads …
2024-07-12 12:51:35 UTC – Dumping uploads … done
2024-07-12 12:51:35 UTC – Dumping builds …
2024-07-12 12:51:35 UTC – Dumping builds … done
2024-07-12 12:51:35 UTC – Dumping artifacts …
2024-07-12 12:51:35 UTC – Dumping artifacts … done
2024-07-12 12:51:35 UTC – Dumping pages …
2024-07-12 12:51:35 UTC – Dumping pages … done
2024-07-12 12:51:35 UTC – Dumping lfs objects …
2024-07-12 12:51:35 UTC – Dumping lfs objects … done
2024-07-12 12:51:35 UTC – Dumping terraform states …
2024-07-12 12:51:35 UTC – Dumping terraform states … done
2024-07-12 12:51:35 UTC – Dumping container registry images … [DISABLED]
2024-07-12 12:51:35 UTC – Dumping packages …
2024-07-12 12:51:35 UTC – Dumping packages … done
2024-07-12 12:51:35 UTC – Dumping ci secure files …
2024-07-12 12:51:35 UTC – Dumping ci secure files … done
2024-07-12 12:51:35 UTC – Creating backup archive: 1720788502_2024_07_12_16.10.1_gitlab_backup.tar …
2024-07-12 12:52:01 UTC – Creating archive 1720788502_2024_07_12_16.10.1_gitlab_backup.tar failed
2024-07-12 12:52:01 UTC – Deleting tar staging files …
log writing failed. No space left on device @ io_write - /opt/gitlab/embedded/service/gitlab-rails/log/backup_json.log
2024-07-12 12:52:01 UTC – Cleaning up /var/opt/gitlab/backups/backup_information.yml
log writing failed. No space left on device @ io_write - /opt/gitlab/embedded/service/gitlab-rails/log/backup_json.log
2024-07-12 12:52:01 UTC – Cleaning up /var/opt/gitlab/backups/db
2024-07-12 12:52:01 UTC – Cleaning up /var/opt/gitlab/backups/repositories
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/uploads.tar.gz
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/builds.tar.gz
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/artifacts.tar.gz
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/pages.tar.gz
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/lfs.tar.gz
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/packages.tar.gz
2024-07-12 12:52:02 UTC – Cleaning up /var/opt/gitlab/backups/ci_secure_files.tar.gz
2024-07-12 12:52:02 UTC – Deleting tar staging files … done
2024-07-12 12:52:02 UTC – Deleting backups/tmp …
2024-07-12 12:52:02 UTC – Deleting backups/tmp … done
2024-07-12 12:52:02 UTC – Deleting backup and restore PID file … done
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:260:in block in pack' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:240:in chdir’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:240:in pack' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:147:in run_all_create_tasks’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:28:in create' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in block in create_backup’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:68:in lock_backup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in create_backup’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:107:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in block in execute’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in each' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in execute’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in execute' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in block in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in synchronize' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in invoke_with_call_chain’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in invoke' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in invoke_task’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block (2 levels) in top_level' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in each’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in block in top_level' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in run_with_threads’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in top_level' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in block in run’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in standard_exception_handling' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in run’
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>' /opt/gitlab/embedded/bin/rake:25:in load’
/opt/gitlab/embedded/bin/rake:25:in <top (required)>' /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in load’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in kernel_load' /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in run’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:451:in exec' /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:28:in run’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command' /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:527:in dispatch’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:34:in dispatch' /opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:584:in start’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:28:in start' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/exe/bundle:28:in block in <top (required)>’
/opt/gitlab/embedded/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/bundler-2.5.6/exe/bundle:20:in <top (required)>’
/opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in ’
Tasks: TOP => gitlab:backup:create

2

NOTE the log tell I don’t have enough space to write to log file… but I known I have sufficient disk space… does it possible Gitlab calculate some extra free space to ensure a minimum free still available ?

3

Can you post the results from connecting to the server via ssh and doing:

df -h

and also:

df -i
4

Hi, sorry I didn’t have a copy of “df -h”… But I had 12G free on my /dev/sda1 (where the backups folder exist).

I decided to grow my hard drive partition using GParted LiveCD (offline) to add 40G.

Then, I launch again a “gitlab-backup” and it was successful now ! But like I suspected, the backup.tar was a 8.7G file size and I had 12G free space. It should be enough because my last backup in june when I make another upgrade was using nearly the same space disk.

Anyway, It confirm I need some extra free space to create the backup TAR file.

Regards