Error reconfiguring new system with old configuration

yeray_alonso · September 30, 2024, 9:47pm

While trying to prepare a new server for a migration from an old system, I do:

Update old server
Install Ubuntu Server 24.04.1 on the new server
Install gitlab-ce
Copy gitlab.rb and gitlab-secrets.json from the old system
Run sudo gitlab-ctl reconfigure

This is the error I get:

Generated at 2024-09-30 21:06:48 +0000
Mixlib::ShellOut::ShellCommandFailed: storage_directory[/home/administrator/lfs-objects] (gitlab::gitlab-rails line 64) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[directory resource: /home/administrator/lfs-objects] (gitlab::gitlab-rails line 36) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of stat --printf='%U' $(readlink -f /home/administrator/lfs-objects) ----
STDOUT: 
STDERR: stat: missing operand
Try 'stat --help' for more information.
---- End output of stat --printf='%U' $(readlink -f /home/administrator/lfs-objects) ----
Ran stat --printf='%U' $(readlink -f /home/administrator/lfs-objects) returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.8/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:35:in `run_command'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:67:in `get_owner'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:48:in `ensure_permissions_set'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb:42:in `block (3 levels) in class_from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/ruby_block.rb:31:in `block (2 levels) in <class:RubyBlock>'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider/ruby_block.rb:30:in `block in <class:RubyBlock>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/dsl/resources.rb:36:in `ruby_block'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb:36:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:860:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:855:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:855:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:879:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/client.rb:298:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application/base.rb:354:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-18.3.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.3.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'

Am I missing anything obvious?

yeray_alonso · October 1, 2024, 5:53am

Commenting the following lines sudo gitlab-ctl reconfigure ends successfully for me:

#gitlab_rails['lfs_enabled'] = true
#gitlab_rails['lfs_storage_path'] = "/home/administrator/lfs-objects"

I also had to comment all the mattermost related lines to skip another error:

#mattermost['gitlab_enable'] = true
#mattermost['gitlab_secret'] = "..."
#mattermost['gitlab_id'] = "..."
#mattermost['gitlab_scope'] = ""
#mattermost['gitlab_auth_endpoint'] = "..."
#mattermost['gitlab_token_endpoint'] = "..."
#mattermost['gitlab_user_api_endpoint'] = "..."

#mattermost['email_enable_sign_up_with_email'] = false
#mattermost['email_enable_sign_in_with_email'] = false

yeray_alonso · October 1, 2024, 6:50am

Without lfs and mattermost I could complete the migration to the new server without errors.
However, as soon as I try to re-enable lfs with those two gitlab_rails['lfs_*'] lines the reconfiguration fails again with the same error.

I’ve forced the /home/administrator/lfs-objects folder ownership to be from the git user and group (to leave it as it was in the old server) with sudo chown git:git /home/administrator/lfs-objects, but the reconfiguration gives the same error.

dnsmichi · October 1, 2024, 7:48pm

This directory sounds like root user on Linux/Unix, where other users cannot change into, no matter the underlying directories and their permissions.

sudo -u git ls -la /home/administrator/ should give insights, or any other way of becoming the git user, and checking file access.

Is there a specific reason to not use the default LFS path? GitLab Git Large File Storage (LFS) Administration | GitLab

yeray_alonso · October 2, 2024, 10:13am

Indeed, it doesn’t let the git user to access that folder:

ls: cannot open directory ‘/home/administrator/’: Permission denied

That’s how it was set in the old server, but I’m not sure why.
Since lfs seems to be enabled by default I guess I can leave those lines commented.

Thanks!

dnsmichi · October 2, 2024, 3:06pm

Keep note of the default location when migrating the data assets (Git repos, LFS storage) from the old server.

yeray_alonso · October 2, 2024, 3:55pm

This is what seems to be in the old server

imagen

Which looks exactly the same as in the default location of the new server:

So I guess the backup and restore process already moved everything (not much though).

dnsmichi · October 2, 2024, 5:09pm

Ah ok, it worked seaminglessly. Thanks for sharing - I have never done that procedure myself with LFS