Branch diff times out: How can the timeout be raised?

Hello,

when comparing big branches (they are in git-lfs), the process seems to time out and we get a “500 Whoops, something went wrong on our end.” error. In /var/log/gitlab/gitlab-rails/production.log we find the following, probably related, entries:

Completed 500 Internal Server Error in 30833ms (ActiveRecord: 58.7ms | Elasticsearch: 0.0ms)
ActionView::Template::Error (4:Deadline Exceeded):
1: - environment = local_assigns.fetch(:environment, nil)
2: - show_whitespace_toggle = local_assigns.fetch(:show_whitespace_toggle, true)
3: - can_create_note = !@diff_notes_disabled && can?(current_user, :create_note, diffs.project)
4: - diff_files = diffs.diff_files
5: - diff_page_context = local_assigns.fetch(:diff_page_context, nil)
6:
7: .content-block.oneline-block.files-changed.diff-files-changed.js-diff-files-changed
lib/gitlab/gitaly_client/diff_stitcher.rb:15:in `each'
lib/gitlab/gitaly_client/diff_stitcher.rb:15:in `each'
lib/gitlab/git/diff_collection.rb:115:in `each_gitaly_patch'
lib/gitlab/metrics/instrumentation.rb:161:in `block in each_gitaly_patch'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `each_gitaly_patch'
lib/gitlab/git/diff_collection.rb:49:in `each'
lib/gitlab/metrics/instrumentation.rb:161:in `block in each'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `each'
lib/gitlab/git/diff_collection.rb:91:in `each_with_index'
lib/gitlab/git/diff_collection.rb:91:in `decorate!'
lib/gitlab/metrics/instrumentation.rb:161:in `block in decorate!'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `decorate!'
lib/gitlab/diff/file_collection/base.rb:34:in `diff_files'
app/views/projects/diffs/_diffs.html.haml:4:in `_app_views_projects_diffs__diffs_html_haml___1987965041244661414_70020606383200'
app/views/projects/compare/show.html.haml:9:in `_app_views_projects_compare_show_html_haml__3254105502642255833_70020603069800'
app/controllers/application_controller.rb:118:in `render'
app/controllers/projects/compare_controller.rb:28:in `show'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

This looks like a 30s timeout to me. But I might be wrong.

If it is a timeout problem: Can this timeout be raised? I cannot find anything appropriate in the Web UI or in /etc/gitlab/gitlab.rb. Or if this isn’t a timeout problem, what could it be otherwise?

We are running gitlab-ce 12.3.5-ce.0 on Debian 8.11, installed and regularly updated using apt from https://packages.gitlab.com/gitlab/gitlab-ce/debian/.

Thanks
frank

Seeing the same issue, this is a large diff which takes about 37 seconds to process using git diff locally

PS C:\Users\bagdon\repo> Measure-Command { git diff 2995301778da302991bddf7a6866e17d18bd8c5c f4495b347fb56019ffc4148994dc9bb617e833bc }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 37
Milliseconds      : 432
Ticks             : 374321610
TotalDays         : 0.000433242604166667
TotalHours        : 0.0103978225
TotalMinutes      : 0.62386935
TotalSeconds      : 37.432161
TotalMilliseconds : 37432.161

Stacktrace:

Completed 500 Internal Server Error in 23009ms (ActiveRecord: 7.3ms | Elasticsearch: 0.0ms)
ActionView::Template::Error (4:Deadline Exceeded):
1: - environment = local_assigns.fetch(:environment, nil)
2: - diff_page_context = local_assigns.fetch(:diff_page_context, nil)
3: - file_hash = hexdigest(diff_file.file_path)
4: - image_diff = diff_file.rich_viewer && diff_file.rich_viewer.partial_name == 'image'
5: - image_replaced = diff_file.old_content_sha && diff_file.old_content_sha != diff_file.content_sha
6:
7: .diff-file.file-holder{ id: file_hash, data: diff_file_html_data(project, diff_file.file_path, diff_file.content_sha) }
lib/gitlab/gitaly_client/blobs_stitcher.rb:15:in `each'
lib/gitlab/gitaly_client/blobs_stitcher.rb:15:in `each'
lib/gitlab/git/blob.rb:70:in `to_a'
lib/gitlab/git/blob.rb:70:in `batch'
lib/gitlab/git/repository.rb:823:in `batch_blobs'
app/models/repository.rb:520:in `blobs_at'
lib/gitlab/metrics/instrumentation.rb:161:in `block in blobs_at'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `blobs_at'
app/models/blob.rb:85:in `block in lazy'
lib/gitlab/diff/file.rb:143:in `block in old_blob'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
lib/gitlab/diff/file.rb:142:in `old_blob'
lib/gitlab/diff/file.rb:380:in `blobs_changed?'
lib/gitlab/diff/file.rb:265:in `content_changed?'
lib/gitlab/diff/file.rb:431:in `viewer_class_from'
lib/gitlab/diff/file.rb:426:in `rich_viewer_class'
lib/gitlab/diff/file.rb:314:in `rich_viewer'
app/views/projects/diffs/_file.html.haml:4:in `_app_views_projects_diffs__file_html_haml___1958047491488583970_69812883557700'
app/views/projects/diffs/_diffs.html.haml:28:in `_app_views_projects_diffs__diffs_html_haml__3145270096785188552_69813044769320'
app/views/projects/commit/show.html.haml:14:in `_app_views_projects_commit_show_html_haml___622432748504418306_69813186274160'
app/controllers/application_controller.rb:108:in `render'
app/controllers/projects/commit_controller.rb:29:in `block (2 levels) in show'
app/controllers/projects/commit_controller.rb:27:in `show'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:62:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:48:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:32:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Using gitlab omnibus 12.5.2