Gitlab-ctl reconfigure fail with 10.3 update

How to Use GitLab
1

Hello,

i just updated our omnibus install from 10.2.4 to 10.3.2 on ubuntu 14.04 and the Gitlab-ctl reconfigure failed leading to error 500.

I managed to make it succed by commenting the lines about backups :

 gitlab_rails['backup_path'] = "/media/comp/DATAS/backups"
 gitlab_rails['backup_archive_permissions'] = 0644
 gitlab_rails['backup_keep_time'] = 259200

But i’d rather keep it like this in order to have my backup working as usual.

This is the output of the reconfigure if this can help :

sudo gitlab-ctl reconfigure
Starting Chef Client, version 12.12.15
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - package (0.1.0)
  - registry (0.1.0)
  - mattermost (0.1.0)
  - consul (0.0.0)
  - gitaly (0.1.0)
  - runit (0.14.2)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: gitlab::default
  * directory[/etc/gitlab] action create (up to date)
  Converging 408 resources
  * directory[/etc/gitlab] action create (up to date)
  * directory[Create /var/opt/gitlab] action create (up to date)
  * directory[/opt/gitlab/embedded/etc] action create (up to date)
  * template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
  * account[Webserver user and group] action create
    * group[Webserver user and group] action create (up to date)
    * user[Webserver user and group] action create (up to date)
     (up to date)
Recipe: gitlab::users
  * directory[/var/opt/gitlab] action create (up to date)
  * account[GitLab user and group] action create
    * group[GitLab user and group] action create (up to date)
    * user[GitLab user and group] action create (up to date)
     (up to date)
  * template[/var/opt/gitlab/.gitconfig] action create (up to date)
Recipe: gitlab::gitlab-shell
  * storage_directory[/var/opt/gitlab/git-data] action create
    * ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
     (up to date)
  * storage_directory[/var/opt/gitlab/git-data/repositories] action create
    * ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run (skipped due to not_if)
     (up to date)
  * storage_directory[/var/opt/gitlab/.ssh] action create
    * ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)
     (up to date)
  * directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
  * templatesymlink[Create a config.yml and create a symlink to Rails root] action create
    * template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
    * link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
     (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
  * execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] action run
    - execute /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions
  * bash[Set proper security context on ssh files for selinux] action run (skipped due to only_if)
Recipe: gitlab::gitlab-rails
  * directory[/var/log/gitlab] action create (up to date)
  * storage_directory[/var/opt/gitlab/gitlab-rails/shared] action create
    * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run (skipped due to not_if)
     (up to date)
  * storage_directory[/var/opt/gitlab/gitlab-rails/shared/artifacts] action create
    * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run (skipped due to not_if)
     (up to date)
  * storage_directory[/var/opt/gitlab/gitlab-rails/shared/lfs-objects] action create
    * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run (skipped due to not_if)
     (up to date)
  * storage_directory[/var/opt/gitlab/gitlab-rails/uploads] action create
    * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run (skipped due to not_if)
     (up to date)
  * storage_directory[/var/opt/gitlab/gitlab-ci/builds] action create
    * ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run (skipped due to not_if)
     (up to date)
  * storage_directory[/var/opt/gitlab/gitlab-rails/shared/pages] action create
    * ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run (skipped due to not_if)
     (up to date)
  * directory[create /var/opt/gitlab/gitlab-rails/etc] action create (up to date)
  * directory[create /opt/gitlab/etc/gitlab-rails] action create (up to date)
  * directory[create /var/opt/gitlab/gitlab-rails/working] action create (up to date)
  * directory[create /var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
  * directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
  * directory[create /var/log/gitlab/gitlab-rails] action create (up to date)
  * storage_directory[/media/comp/DATAS/backups] action create
    * ruby_block[directory resource: /media/comp/DATAS/backups] action run
      
      ================================================================================
      Error executing action `run` on resource 'ruby_block[directory resource: /media/comp/DATAS/backups]'
      ================================================================================
      
      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received '1'
      ---- Begin output of stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) ----
      STDOUT: 
      STDERR: stat: opérande manquant
      Saisissez « stat --help » pour plus d'informations.
      ---- End output of stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) ----
      Ran stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) returned 1
      
      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:35:in `run_command'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:67:in `get_owner'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:48:in `ensure_permissions_set'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb:39:in `block (3 levels) in class_from_file'
      
      Resource Declaration:
      ---------------------
      # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb
      
       33:   ruby_block "directory resource: #{path}" do
       34:     block do
       35:       # Ensure the directory exists
       36:       storage_helper.ensure_directory_exists(path)
       37: 
       38:       # Ensure the permissions are set
       39:       storage_helper.ensure_permissions_set(path)
       40: 
       41:       # Error out if we have not achieved the target permissions
       42:       storage_helper.validate!(path)
       43:     end
       44:     not_if { storage_helper.validate(path) }
       45:   end
       46: end
      
      Compiled Resource:
      ------------------
      # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb:33:in `block in class_from_file'
      
      ruby_block("directory resource: /media/comp/DATAS/backups") do
        action [:run]
        retries 0
        retry_delay 2
        default_guard_interpreter :default
        block_name "directory resource: /media/comp/DATAS/backups"
        declared_type :ruby_block
        cookbook_name "gitlab"
        block #<Proc:0x0000000003b74d48@/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb:34>
        not_if { #code block }
      end
      
      Platform:
      ---------
      x86_64-linux
      
    
    ================================================================================
    Error executing action `create` on resource 'storage_directory[/media/comp/DATAS/backups]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    ruby_block[directory resource: /media/comp/DATAS/backups] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb line 33) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
    ---- Begin output of stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) ----
    STDOUT: 
    STDERR: stat: opérande manquant
    Saisissez « stat --help » pour plus d'informations.
    ---- End output of stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) ----
    Ran stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) returned 1
    
    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:35:in `run_command'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:67:in `get_owner'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/storage_directory_helper.rb:48:in `ensure_permissions_set'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb:39:in `block (3 levels) in class_from_file'
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
    
     95: storage_directory node['gitlab']['gitlab-rails']['backup_path'] do
     96:   owner gitlab_user
     97:   mode '0700'
     98:   only_if { node['gitlab']['gitlab-rails']['manage_backup_path'] }
     99: end
    100: 
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:95:in `from_file'
    
    storage_directory("/media/comp/DATAS/backups") do
      action [:create]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      declared_type :storage_directory
      cookbook_name "gitlab"
      recipe_name "gitlab-rails"
      owner "git"
      mode "0700"
      path "/media/comp/DATAS/backups"
      only_if { #code block }
    end
    
    Platform:
    ---------
    x86_64-linux
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

storage_directory[/media/comp/DATAS/backups] (gitlab::gitlab-rails line 95) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[directory resource: /media/comp/DATAS/backups] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb line 33) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) ----
STDOUT: 
STDERR: stat: opérande manquant
Saisissez « stat --help » pour plus d'informations.
---- End output of stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) ----
Ran stat --printf='%U' $(readlink -f /media/comp/DATAS/backups) returned 1

Running handlers complete
Chef Client failed. 1 resources updated in 04 seconds

The backups folder is set to 0777 for the sake of the test.

Any idea what’s going on ? It’s the first time an update failed like that in almost 2 years.

Thanks

2

Got the same problem. Reconfigure failed due to backup permissions…

We us an nfs mount on /mnt/backups and is has 755 permissions. Moving backup to a local dir with 700 worked. Whats up with that? 80+ collegues had no gitlab. It does not even write anything into the dir!

The error we get is:

Running handlers:
There was an error running gitlab-ctl reconfigure:

storage_directory[/mnt/backups] (gitlab::gitlab-rails line 95) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[directory resource: /mnt/backups] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/storage_directory.rb line 33) had an error: Mixlib::ShellOut::ShellCommandFailed: Failed asserting that mode permissions on "/mnt/backups" is 0700
---- Begin output of set -x && [ "$(stat --printf='%04a' $(readlink -f /mnt/backups) | grep -o '....$')" = '0700' ] ----
STDOUT:
STDERR: + grep -o ....$
+ readlink -f /mnt/backups
+ stat --printf=%04a /mnt/backups
+ [ 0755 = 0700 ]
---- End output of set -x && [ "$(stat --printf='%04a' $(readlink -f /mnt/backups) | grep -o '....$')" = '0700' ] ----
Ran set -x && [ "$(stat --printf='%04a' $(readlink -f /mnt/backups) | grep -o '....$')" = '0700' ] returned 1
3

I manage to solve this by setting the user git as owner of the full path.

in my case the path was : /media/comp/DATAS/backups and “comp” was not owned by git. Changing that with a chown allowed me to run reconfigure without any error.

It always works like this before this version , i dont know exactly what changed.

4

Found out wat specifically was the issue:

Within the gitlab.rb the setting

gitlab_rails['manage_backup_path'] = true

was set. Due to this setting the recent gitlab migration scripts try to ensure restrictive owner permissions of 700 to the backup dir. As our dir is mounted with 755 on an NFS mount this set fails.

Due to the fail of that step all further reconfigure scripts are omitted and half the system stays in the old configuration.

In my opinion this should not happen! The seeting wordked before and it would be legitimite to just give a warning (probably saying) disable the manage option in the settings and continue rather than failing the whole process just because a permission (which is actually not needed to perform the reconfigure!!) failed to being set.

Setting

gitlab_rails['manage_backup_path'] = false

solves the issue obviously and is correct in our case. But the fail still isn’t ok in my opinion. A non critical operation not necessarily needed for operation should not break a whole gitlab installation.