502 (too much time to respond) only on repo-related actions

How to Use GitLab
1

My GitLab installation was working perfectly about a month ago, but now it is showing 502 GitLab is taking too much time to respond error pages. The interesting thing is, it is only showing them for git-repo related write actions:

502 occurs on:

  • Create/delete branch
  • Create/edit file
  • Merging merge requests

Working fine:

  • Create/Update Issue
  • Create/update project
  • Create/Change Merge Request (but not merging)
  • Create/edit snippet
  • Pushing changes from local pc

The last one is also weird. Repo changes work perfectly fine when pushing, but not via the webinterface.

My Setup

GitLab is installed via docker (gitlab/gitlab-ce:latest v8.16.4-ce.0)

GitLab 8.16.4
GitLab Shell 4.1.1
GitLab Workhorse v1.3.0
GitLab API v3
Git 2.10.2
Ruby 2.3.3p222
Rails 4.2.7.1
PostgreSQL 9.2.18

What I’ve tried

  • Update (newest version, still has problem)
  • Restart, clean caches, …
  • gitlab-rake gitlab:check: everything fine
  • docker exec gitlab update-permissions: output

Logs

I don’t find very promising logs, but this is everything from gitlab-ctl tail during a branch creation attempt:

 ==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/manuel/test-vpdb/branches" for 62.48.213.246 at 2017-02-02 16:45:38 +0000
Processing by Projects::BranchesController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"VDQZ+WekTdOcNG7u1QOM9CM6Yt6adGA5R2qOsYguKeV/eC+F10BYStMqdhlIpalo/POWm5sWJGtxLgtcSmV/dg==", "branch_name"=>"testbranch", "ref"=>"master", "namespace_id"=>"manuel", "project_id"=>"test-vpdb"}

==> /var/log/gitlab/redis/current <==
2017-02-02_16:46:01.74361 384:M 02 Feb 16:46:01.743 * 10000 changes in 60 seconds. Saving...
2017-02-02_16:46:01.74383 384:M 02 Feb 16:46:01.743 * Background saving started by pid 1003
2017-02-02_16:46:01.90762 1003:C 02 Feb 16:46:01.907 * DB saved on disk
2017-02-02_16:46:01.90823 1003:C 02 Feb 16:46:01.908 * RDB: 0 MB of memory used by copy-on-write
2017-02-02_16:46:01.94447 384:M 02 Feb 16:46:01.944 * Background saving terminated with success

==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
E, [2017-02-02T16:46:09.441430 #430] ERROR -- : worker=0 PID:742 timeout (31s > 30s), killing
E, [2017-02-02T16:46:09.467144 #430] ERROR -- : reaped #<Process::Status: pid 742 SIGKILL (signal 9)> worker=0

==> /var/log/gitlab/gitlab-workhorse/current <==
2017-02-02_16:46:09.46742 2017/02/02 16:46:09 error: POST "/manuel/test-vpdb/branches": badgateway: failed after 31s: EOF
2017-02-02_16:46:09.46749 2017/02/02 16:46:09 ErrorPage: serving predefined error page: 502
2017-02-02_16:46:09.46762 gitlab.x.org @ - - [2017-02-02 16:45:38.48315806 +0000 UTC] "POST /manuel/test-vpdb/branches HTTP/1.1" 502 2662 "http://gitlab.x.org/manuel/test-vpdb/branches/new" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" 30.984423

==> /var/log/gitlab/nginx/gitlab_access.log <==
62.48.213.246 - - [02/Feb/2017:16:46:09 +0000] "POST /manuel/test-vpdb/branches HTTP/1.1" 502 2674 "http://gitlab.x.org/manuel/test-vpdb/branches/new" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"

Any help welcome