502 - GitLab 7.4.3 is not responding

Gitlab was working perfectly since while. We are using version - 7.4.3, Omnibus installation.

It started with 502 - Gatlab is not responding error.

I check

sudo gitlab-ctl status

run: logrotate: (pid 1017) 1880s; run: log: (pid 1010) 1880s
run: nginx: (pid 1030) 1880s; run: log: (pid 1014) 1880s
run: postgresql: (pid 1022) 1880s; run: log: (pid 1011) 1880s
run: redis: (pid 1019) 1880s; run: log: (pid 1012) 1880s
run: sidekiq: (pid 17080) 2s; run: log: (pid 1028) 1880s
run: unicorn: (pid 17071) 2s; run: log: (pid 1023) 1880s

If i try to have backup by

sudo gitlab-rake gitlab:backup:create

It is throwing me errors as below (i tried with --trace as well)

rake aborted!
LoadError: cannot load such file -- fog/aws/region_methods
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:232:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-1.21.0/lib/fog/aws/core.rb:5:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:232:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-1.21.0/lib/fog/aws/auto_scaling.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:232:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-1.21.0/lib/fog/aws.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:232:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/fog-1.21.0/lib/fog.rb:22:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:232:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/carrierwave-0.9.0/lib/carrierwave/storage/fog.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:232:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/carrierwave-0.9.0/lib/carrierwave/storage.rb:9:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:232:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.1/lib/active_support/dependencies.rb:247:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/carrierwave-0.9.0/lib/carrierwave.rb:71:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:76:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:76:in `block (2 levels) in require'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:72:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:72:in `block in require'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:61:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:61:in `require'
/opt/gitlab/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler.rb:131:in `require'
/opt/gitlab/embedded/service/gitlab-rails/config/application.rb:6:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/Rakefile:5:in `require'
/opt/gitlab/embedded/service/gitlab-rails/Rakefile:5:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/rake_module.rb:28:in `load'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/application.rb:94:in `block in load_rakefile'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/application.rb:93:in `load_rakefile'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/application.rb:77:in `block in run'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/lib/rake/application.rb:75:in `run'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/rake-10.3.2/bin/rake:33:in `<top (required)>'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/rake:23:in `load'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/rake:23:in `<main>'

To check the status of internals and i search on many other places. I observed that; it is throwing above error whenever i use any command using - gitlab-rake

Let me know if i require to add any log file.
P.S. - We are not pro in ruby.

Can you try to do a gitlab-ctl reconfigure?

Yes I did so. It says recinfirgure successful. 2 files updated. But no result in expectation.

Apartfrom above I did below things so far to fix
Problem1: it was showing Sidekiq & Unicorn timeout in status.
Solution:. I deleted socket file, made sure it have enough memory & disk space. Killed process manually and restart Gitlab.

Any update on this? My team is stuck as repository is completely down.

Please help

According to this issue, AWS changed the way it handles regions

https://github.com/gitlabhq/gitlabhq/issues/8402

Try to leave out the region section to see if that fixes your error.

As for the 502 error, without providing some logs there is nothing much we can do to help. For starters, you can try upgrading to latest version.

Here is error log. Please do let me know if you want me any to send specific to any log file.

I am bit scared to upgrade to new version. We might break some more things.

/var/log/gitlab/nginx/gitlab_error.log



    2015/02/23 16:32:23 [crit] 1062#0: *3 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file o$
2015/02/23 16:32:25 [crit] 1062#0: *5 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file o$
2015/02/23 16:33:10 [crit] 1062#0: *8 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file o$
2015/02/23 16:42:23 [crit] 1062#0: *10 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file $
2015/02/23 16:52:20 [crit] 1062#0: *12 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file $
2015/02/23 16:57:06 [crit] 1062#0: *14 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file $
2015/02/23 17:02:26 [crit] 1062#0: *16 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file $
2015/02/23 17:12:28 [crit] 1062#0: *18 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file $
2015/02/23 17:17:40 [crit] 1062#0: *20 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (2: No such file $

From one more file

==> /var/log/gitlab/unicorn/current <==



2015-02-23_10:49:58.25565 starting new unicorn master
2015-02-23_10:50:17.77047 master failed to start, check stderr log for details
2015-02-23_10:50:18.77425 failed to start a new unicorn master
2015-02-23_10:50:18.77814 starting new unicorn master
2015-02-23_10:50:21.27380 master failed to start, check stderr log for details
2015-02-23_10:50:22.27747 failed to start a new unicorn master
2015-02-23_10:50:22.28164 starting new unicorn master
2015-02-23_10:50:24.76055 master failed to start, check stderr log for details
2015-02-23_10:50:25.76428 failed to start a new unicorn master
2015-02-23_10:50:25.76820 starting new unicorn master
2015-02-23_10:50:28.22377 master failed to start, check stderr log for details
2015-02-23_10:50:29.22773 failed to start a new unicorn master
2015-02-23_10:50:29.23164 starting new unicorn master
2015-02-23_10:50:31.65201 master failed to start, check stderr log for details
2015-02-23_10:50:32.65593 failed to start a new unicorn master
2015-02-23_10:50:32.66031 starting new unicorn master
2015-02-23_10:50:35.08719 master failed to start, check stderr log for details
2015-02-23_10:50:36.09114 failed to start a new unicorn master
2015-02-23_10:50:36.09514 starting new unicorn master
2015-02-23_10:50:38.50441 master failed to start, check stderr log for details
2015-02-23_10:50:39.50825 failed to start a new unicorn master
2015-02-23_10:50:39.51220 starting new unicorn master
2015-02-23_10:50:41.94969 master failed to start, check stderr log for details
2015-02-23_10:50:42.95353 failed to start a new unicorn master
2015-02-23_10:50:42.95777 starting new unicorn master
2015-02-23_10:50:45.39573 master failed to start, check stderr log for details
2015-02-23_10:50:46.39963 failed to start a new unicorn master
2015-02-23_10:50:46.40354 starting new unicorn master
2015-02-23_10:50:48.84316 master failed to start, check stderr log for details
2015-02-23_10:50:49.84731 failed to start a new unicorn master
2015-02-23_10:50:49.85126 starting new unicorn master
2015-02-23_10:50:52.30194 master failed to start, check stderr log for details
2015-02-23_10:50:53.30597 failed to start a new unicorn master
2015-02-23_10:50:53.30985 starting new unicorn master
2015-02-23_10:50:55.71552 master failed to start, check stderr log for details
2015-02-23_10:50:56.71937 failed to start a new unicorn master

Seems unicorn fails to create the socket. Try again:

gitlab-ctl reconfigure
gitlab-ctl restart

Do you have any special configuration in /etc/gitlab/gitlab.rb? Can you paste its content, leaving out any private information?

gitlab-ctl reconfigure

Chef Client finished, 3/147 resources updated in 23.591263049 seconds

gitlab-ctl restart

ok: run: logrotate: (pid 11365) 1s
ok: run: nginx: (pid 11368) 0s
ok: run: postgresql: (pid 11381) 0s
ok: run: redis: (pid 11393) 1s
ok: run: sidekiq: (pid 11398) 0s
ok: run: unicorn: (pid 11402) 0s

Still no updates on condition. It shows UI as per attachment.

One Clue - I am trying to search .socket file mentioned in log; but I found it is not been created.

Content of gitlab.rb

external_url = 'URL'
gitlab_rails['gitlab_email_from'] = 'EMAIL'
a = sign
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.mandrillapp.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "USERNAME"
gitlab_rails['smtp_password'] = "PASSWORD"
gitlab_rails['smtp_domain'] = "DOMAIN"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true

Can you try and remove the equal sign = ?

It should look like:

external_url 'URL'

Then reconfigure and see if that fixes it.

Also what is that?

I got this issue fixed by upgrading to latest version. It was very quick. I should have done the long before as @axil suggested. Thank you very much; you guys are awesome so as Gitlab.

Check your unicorn for more details. I had the same problem, when I was installing gitlab alongside jenkins, and there was a conflict between jenkins default 8080 port and unicorn 8080 port

2 Likes