History on some file get 504 Gateway time-out

When I’m trying to get some history for a given file, some files get a 504 Gateway time-out

I can understand that computing history can be costly but is there a way to fix it?

Actually our repo is about 5k commits, and all files are not impacted.

Logs I found

gitlab_1  | ==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
gitlab_1  | E, [2017-04-21T16:30:20.621789 #483] ERROR -- : worker=5 PID:575 timeout (61s > 60s), killing
gitlab_1  |
gitlab_1  | ==> /var/log/gitlab/gitlab-workhorse/current <==
gitlab_1  | 2017-04-21_16:30:20.63682 2017/04/21 16:30:20 error: GET "/PATH/TO/THE/file": badgateway: failed after 61s: EOF
gitlab_1  | 2017-04-21_16:30:20.65855 2017/04/21 16:30:20 ErrorPage: serving predefined error page: 502
gitlab_1  | 2017-04-21_16:30:20.66441 gitlab.hostname @ - - [2017-04-21 16:29:19.454024696 +0000 UTC] "GET /PATH/TO/THE/file HTTP/1.1" 502 2662 "https://gitlab.hostname/group/project/blob/master/PATH/TO/THE/file" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" 61.210357
gitlab_1  |
gitlab_1  | ==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
gitlab_1  | E, [2017-04-21T16:30:20.685724 #483] ERROR -- : reaped #<Process::Status: pid 575 SIGKILL (signal 9)> worker=5
gitlab_1  | I, [2017-04-21T16:30:20.784468 #2577]  INFO -- : worker=5 ready

Here gitlab:check result

Checking GitLab Shell ...

GitLab Shell version >= 5.0.0 ? ... OK (5.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
ALL REPOS ARE OK
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
ALL REPOS ARE YES
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 27

Checking GitLab ... Finished

When I try with rails console

Loading production environment (Rails 4.2.8)
irb(main):001:0> app.get 'https://gitlab.hostname/NAMESPACE/PROJECT/commits/BRANCH/PATH/TO/THE/file?private_token=aN4amPvSqPBpcyY2GyfX'
Started GET "/NAMESPACE/PROJECT/commits/BRANCH/PATH/TO/THE/file?private_token=[FILTERED]" for 127.0.0.1 at 2017-04-24 08:50:38 +0000
Processing by Projects::CommitsController#show as HTML
  Parameters: {"private_token"=>"[FILTERED]", "namespace_id"=>"NAMESPACE", "project_id"=>"PROJECT", "id"=>"BRANCH//PATH/TO/THE/file"}
  Cache digest for app/views/projects/commits/_commit.html.haml: 83716d8c1e4e5edc4b0412b055f44e30
Read fragment views/NAMESPACE/PROJECT/55c72afb54e9e7e0752e2212f2e31d7c44cba20d/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (0.2ms)
Read fragment views/NAMESPACE/PROJECT/e1903ff9b69753455b159acc5c489ae86293b329/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (0.2ms)
Read fragment views/NAMESPACE/PROJECT/50f644a2564905960633784ac080676dc4f926bc/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (0.2ms)
Read fragment views/NAMESPACE/PROJECT/f5457ba8a3d7560ce052191d9d462c590b081bb9/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (0.2ms)
Read fragment views/NAMESPACE/PROJECT/8bc1447cd86508d2c71849ba4ca15e18da2fe23e/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (0.2ms)
Read fragment views/NAMESPACE/PROJECT/4671500330dc39338a717b4766618b9a00423941/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (0.2ms)
Read fragment views/NAMESPACE/PROJECT/2213e5b0e5a305eab13b158d4b615d69978c9383/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (0.2ms)
Write fragment views/NAMESPACE/PROJECT/2213e5b0e5a305eab13b158d4b615d69978c9383/application_settings/1-20170202132058486998000/0/83716d8c1e4e5edc4b0412b055f44e30 (14.4ms)
Completed 200 OK in 316452ms (Views: 316141.3ms | ActiveRecord: 101.6ms)
=> 200

It work after 316452ms but it seems really really slow just for history… (where blame is close to instante)


PS: Upgrade to GitLab 9.1.0 does not fix anything and I increase unicorn timeout from 60s to 120s always 504


PS2: Since I force execution with rails console everything seems to be OK now, I think there is some cache somewhere and git command is not launched directly anymore.