Unable to restore GitLab backup successfully

Hi,

My GitLab version is CE 10.1.1.

I am trying to restore the GitLab backup to a new server.

The command to create the backup is “gitlab-rake gitlab:backup:create STRATEGY=copy --trace”.

And the command to restore the backup is “gitlab-backup restore BACKUP=1521884424_2018_03_24_10.5.3”.

But after executing the backup command, it returns the error messages as follows:

TypeError: no implicit conversion of nil into String
/opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:96:in join' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:96:in path_to_repo’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:141:in path_to_repo' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:75:in block in restore’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:73:in restore' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:83:in block (4 levels) in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:56:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => gitlab:backup:repo:restore
(See full trace by running task with --trace)

I have no idea how to resolve the error.
Could you please give me the solution to restore the backup successfully?

Many thanks,
YJ

According to your output above, the backup you are trying to restore is 10.5.3. You can only restore to the exact same version of Gitlab. Therefore, if you want to restore 10.5.3 then gitlab-ce_10.5.3 must be installed on your new server. You cannot restore 10.5.3 to 10.1.1. or any other version number if it doesn’t match.

Also, once gitlab has been installed, you also need to copy gitlab.rb and gitlab-secrets.json from /etc/gitlab on your old server to the new server. Then, once that has been done you run gitlab-ctl reconfigure so that Gitlab and the database is ready with an empty configuration and services running.

Once that part has been done, you can then copy the backup to /var/opt/gitlab/backups and make sure permissions are set with chown git:git /var/opt/gitlab/backups/backup-file.tar so that when you run the restore command, it would be able to access the backup file and restore correctly.

Providing that the gitlab version is the same as the backup, and all the other steps I wrote above, it should restore without any problems whatsoever. It has always worked for me by following those steps completely.

2 Likes

Hi @iwalker,

Thanks for your help.
I typed the wrong version. The correct version I used to restore is 1638294307_2021_12_01_10.1.1_gitlab_backup.tar.

After executing the restoring command, it returns an error while restoring lfs objects:

Put GitLab hooks in repositories dirs [DONE]
done
Restoring uploads ...
done
Restoring builds ...
done
Restoring artifacts ...
done
Restoring pages ...
done
Restoring lfs objects ...
rake aborted!
Errno::EACCES: Permission denied @ rb_file_s_rename - (/data/storage/lfs-objects, /data/storage/lfs.1638792043)
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:46:in `backup_existing_files_dir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:37:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:197:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:lfs:restore
(See full trace by running task with --trace)

The permission of the /data/storage and /data/storage/lfs-object folders are as follows:
drwxr-xr-x 3 root root 25 十二 6 16:14 storage
drwx------ 2 git root 6 十二 6 16:14 lfs-objects

What should I do to resolve the error?

Many thanks,
YJ

Hi,

From what I see on my installation, lfs-objects has git:git so:

chown git:git lfs-objects

should help. You might even have to do the same for /data/storage as well, since a lot of your screenshots show /data/storage/lfs.xxxxxxxx

Hi @iwalker ,

May I know the permissions of your /data/storage and /data/storage/lfs-object folders?

Mine is as follows after executing the “ls -l” command and nothing in the “lfs-objects” folder:
drwxr-xr-x 3 root root 25 十二 6 16:14 storage
drwx------ 2 git root 6 十二 6 16:14 lfs-objects

Many thanks,
YJ

I don’t have these, so not sure what you are using. All my gitlab stuff is under /var/opt/gitlab which is the default on an omnibus install. Did you change it to /data/storage or are you using docker?

Either way, my lfs-objects directory is as I stated with git:git so I suggest changing it and trying again. Also, if /data/storage is the equivalent of /var/opt/gitlab/gitlab-rails, where my lfs-objects directory exists, then you need to set git:git on that as well or git:gitlab-www.

But as I said, if you are using docker, then it could be different. I don’t have docker, so I cannot tell directly, but if it is a case of you just changing the directory location yourself with omnibus, then you’ll need to set similar to what I wrote above.

1 Like

Hi @iwalker ,

Thanks for your help.

I’ve updated the permissions of the folders.

And I also found the GitLab website can be accessed after restoring the backup even there is the “Errno::EACCES: Permission denied” error mentioned above.

But after upgrading from 11.11.8 to 12.0.12 and executing the “gitlab-ctl reconfigure” command, it returns error message as follows:

Starting Chef Client, version 14.13.11
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - redis (0.1.0)
  - postgresql (0.1.0)
  - registry (0.1.0)
  - package (0.1.0)
  - gitlab (0.0.1)
  - monitoring (0.1.0)
  - gitaly (0.1.0)
  - letsencrypt (0.1.0)
  - nginx (0.1.0)
  - runit (4.3.0)
  - acme (4.0.0)
  - crond (0.1.0)
  - mattermost (0.1.0)
  - consul (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
================================================================================

NoMethodError
-------------
undefined method `[]' for nil:NilClass

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:100:in `block in from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:28:in `auto_vivify'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:100:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:23:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:

 93:    def ee_attribute(name, **config)
 94:      config = { ee: true }.merge(config)
 95:      attribute(name, **config)
 96:    end
 97:
 98:    def from_file(_file_path)
 99:      # Allow auto mash creation during from_file call
100>>     Gitlab::ConfigMash.auto_vivify { super }
101:    end
102:
103:    def method_missing(method_name, *arguments) # rubocop:disable Style/MethodMissing
104:      # Give better message for NilClass errors
105:      # If there are no arguments passed, this is a 'GET' call, and if
106:      # there is no matching key in the configuration, then it has not been set (not even to nil)
107:      # and we will output a nicer error above the exception
108:      if arguments.length.zero? && !configuration.key?(method_name)
109:        breaktxt = '=' * 80

System Info:
------------
chef_version=14.13.11
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

undefined method `[]' for nil:NilClass

Running handlers complete
Chef Client failed. 0 resources updated in 02 seconds

And then I keep upgrading from 12.1.17 to 12.9.5 , it returns error messages as follows:

[18:24:32] root@linux_server:~# apt-get install gitlab-ce=12.9.5-ce.0
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 40 not upgraded.
Need to get 0 B/760 MB of archives.
After this operation, 219 MB of additional disk space will be used.
(Reading database ... 296413 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_12.9.5-ce.0_amd64.deb ...
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/git88new.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_12.9.5-ce.0_amd64.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ce_12.9.5-ce.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
[18:26:17] root@linux_server:~#

I have no idea how to resolve the errors.
Could you please give me the solution to upgrade the GitLab successfully?

Many thanks,
YJ

Unfortunately, I have no idea about that. Restore has always worked for me on 12.x, 13.x and 14.x. Hopefully someone else might join in the conversation here and have some ideas on what to do.

Hi @iwalker ,

I deleted the malformed configuration JSON file found at /opt/gitlab/embedded/nodes/git88new.json. And then executing “apt-get install gitlab-ce=12.9.5-ce.0” again, the malformed configuration JSON file will be recreated and GitLab is upgraded successfully.

Many thanks,
Jason

1 Like

Hi @iwalker ,

Do you execute the following commands before restoring the backup:

  • gitlab-ctl stop puma
  • gitlab-ctl stop sidekiq
  • gitlab-ctl stop unicorn
  • gitlab-ctl stop nginx
  • gitlab-ctl status

Many thanks,
YJ

I follow exactly how the documentation states: Back up and restore GitLab | GitLab

sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq

and then continue with the restore. Unicorn doesn’t exist, it was replaced by puma in a particular Gitlab version (I think 13.x but not sure), so depending on which version of gitlab you are running, you either stop puma or unicorn.

1 Like