Gitaly fails to start, does not find nokogiri-1.10.9

How to Use GitLab
1

Hi there,

I’m regularly struggling with gitaly for a while now each time I upgrade.

I was running Gitlab v13.1.1, installed from source. Everything is fine after updating to v13.1.3 (documentation: https://docs.gitlab.com/ee/update/upgrading_from_source.html) but somehow Gitaly refuses to start!

/home/git/gitlab/log/gitaly.log shows:

time="2020-07-07T11:01:09+02:00" level=info msg="Starting Gitaly" version="Gitaly, version 13.1.3"
time="2020-07-07T11:01:09+02:00" level=info msg="clearing disk cache object folder" path=/srv/git/repositories
time="2020-07-07T11:01:09+02:00" level=info msg="moving disk cache object folder to /srv/git/repositories/+gitaly/tmp/diskcache594174313" path=/srv/git/repositories
time="2020-07-07T11:01:09+02:00" level=info msg="disk cache object folder doesn't exist, no need to remove" path=/srv/git/repositories
time="2020-07-07T11:01:09+02:00" level=info msg="Starting file walker for /srv/git/repositories/+gitaly/cache" path=/srv/git/repositories/+gitaly/cache
time="2020-07-07T11:01:09+02:00" level=info msg="Starting file walker for /srv/git/repositories/+gitaly/state" path=/srv/git/repositories/+gitaly/state
time="2020-07-07T11:01:09+02:00" level=info msg="cleared all cache object files in /srv/git/repositories/+gitaly/tmp/diskcache594174313 after 504.205<C2><B5>s" path=/srv/git/repositories
/usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:91:in `block in materialize': Could not find nokogiri-1.10.9 in any of the sources (Bundler::GemNotFound)
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `map!'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/spec_set.rb:85:in `materialize'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/definition.rb:170:in `specs'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/definition.rb:237:in `specs_for'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/definition.rb:226:in `requested_specs'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:108:in `block in definition_method'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/runtime.rb:20:in `setup'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler.rb:107:in `setup'
        from /usr/local/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/setup.rb:20:in `<top (required)>'
        from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
time="2020-07-07T11:01:09+02:00" level=fatal msg="load config" config_path=/home/git/gitaly/config.toml error="exit status 1; stderr: \"\""

I’ve cleaned completely the gems from gitlab and reinstalled. I see that nokogiri-1.10.9 has been installed so why is gitaly not finding it?

I’m stuck not knowing how to move forward :confused:

Thanks for help.

2

Tried to recompile everything with gitaly:

cd /home/git/gitaly
sudo -u git -H rm -rf *
sudo -u git -H git reset --hard origin/master
sudo -u git -H git fetch --all --tags --prune
sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION)
# recreate config.toml based on config.toml.example
sudo -u git -H make

somehow this time it installs many other gems, including nokogiri-1.10.9…

and everything’s fine again :relieved:

1 Like
3

@xperseguers We appreciate you coming back to share your solution here! Thanks!

4

@slee24 did it already, even marked my “solution” as… a solution (in green) :smiley: