Gitlab:backup:restore failed while processing "restoring artifacts"

Hi,

I’m getting an error when trying to restore a gitlab backup on a test environment running ubuntu 18 and gitlab version 11.10.8, it’s just hangs at this:

2019-10-24 16:56:58 +0000 -- Restoring artifacts ...

After a while (1 hour) I ctrl + c and output below:

Restoring artifacts ...
rake aborted!
Errno::EACCES: Permission denied @ rb_file_s_rename - (/data/gitlab_artifacts, /data/artifacts.1510166497)
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:46:in `backup_existing_files_dir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:37:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:159:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:59:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:backup:artifacts:restore

Before I did this modification () I got the error below: (Didn’t get the same error again)

root@gitlab-temp2:~# tail -f -bash: cannot create temp file for here-document: Read-only file system
-bash: cannot create temp file for here-document: Read-only file system

Still hangs at the “2019-10-24 16:56:58 +0000 – Restoring artifacts …” and output below:

2019-10-24 16:56:58 +0000 -- Restoring builds ...
2019-10-24 16:56:58 +0000 -- done
** Invoke gitlab:backup:artifacts:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:artifacts:restore
2019-10-24 16:56:58 +0000 -- Restoring artifacts ...

If I tail the gitlab-ctl logs I can’t see anything out of the ordinary, partial log below:

==> /var/log/gitlab/gitaly/current <==
2019-10-24_16:56:55.19724 time="2019-10-24T16:56:55Z" level=info msg="From /var/opt/gitlab/git-data/repositories/+gitaly/tmp/repo625763927/repo.bundle" grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=CreateRepositoryFromBundle grpc.request.fullMethod=/gitaly.RepositoryService/CreateRepositoryFromBundle grpc.request.glProjectPath=s3digital/opm_toolset grpc.request.glRepository=project-409 grpc.request.repoPath=s3digital/opm_toolset.git grpc.request.repoStorage=default grpc.request.topLevelGroup=s3digital grpc.service=gitaly.RepositoryService grpc.start_time="2019-10-24T16:56:54Z" peer.address=@ span.kind=server system=grpc

And if I “ps -ef | grep gitlab” I can see the processes running but again, no direct issues, logs below:

root      4212  4211  0 16:50 pts/4    00:00:02 tail --follow=name --retry /var/log/gitlab/prometheus/current /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-10-24-12-55-09.log /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/api_json.log /var/log/gitlab/gitlab-rails/sidekiq_exporter.log /var/log/gitlab/gitlab-rails/grpc.log /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/audit_json.log /var/log/gitlab/gitlab-rails/production.log /var/log/gitlab/gitlab-rails/sidekiq.log /var/log/gitlab/postgresql/current /var/log/gitlab/node-exporter/current /var/log/gitlab/redis-exporter/current /var/log/gitlab/gitlab-workhorse/current /var/log/gitlab/redis/current /var/log/gitlab/gitlab-shell/gitlab-shell.log /var/log/gitlab/sidekiq/current /var/log/gitlab/nginx/access.log /var/log/gitlab/nginx/error.log /var/log/gitlab/nginx/gitlab_error.log /var/log/gitlab/nginx/current /var/log/gitlab/nginx/gitlab_access.log /var/log/gitlab/unicorn/unicorn_stderr.log /var/log/gitlab/unicorn/unicorn_stdout.log /var/log/gitlab/unicorn/current /var/log/gitlab/gitlab-monitor/current /var/log/gitlab/postgres-exporter/current /var/log/gitlab/gitaly/current /var/log/gitlab/logrotate/current /var/log/gitlab/alertmanager/current
git       7079  3880  0 16:56 pts/2    00:00:02 tar --unlink-first --recursive-unlink -C /var/opt/gitlab/gitlab-rails/shared/artifacts -xf -
gitlab-+  7420   347  0 16:58 ?        00:00:00 /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitlab/prometheus/prometheus.yml
root     25478   359  0 21:19 ?        00:00:00 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
root     26657 26433  0 21:36 ?        00:00:00 grep --color=auto gitlab

Any help will be appreciated. Or need any additional information.

Thanks.

Hi, I resolved this issue. The main node only had a 250GB SSD installed and other test containers were created and used all the left over disk space, removed the unused containers and it solved the issues.