Hi,
since a few days, we are getting timeouts (Error 502) when eg. comparing 2 branches or scrolling in the commit list. At the beginning even opening the commit list caused a timeout.
Out GitLab Omnibus package is installed on Ubuntu 16.04 LTS. The GitLab version is 10.8.4
In the gitlab-ctl tail you can see this here, when comparing 2 branches ended in an error 502
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
E, [2018-06-08T16:11:01.434508 #1730] ERROR -- : worker=2 PID:6183 timeout (61s > 60s), killing
E, [2018-06-08T16:11:01.465306 #1730] ERROR -- : reaped #<Process::Status: pid 6183 SIGKILL (signal 9)> worker=2
==> /var/log/gitlab/gitlab-workhorse/current <==
2018-06-08_14:11:01.46623 time="2018-06-08T16:11:01+02:00" level=error msg=error error="badgateway: failed after 61s: EOF" method=GET uri="/lxxx/lxxx/compare/lxxx-central%2F0.238-SNAPSHOT...master"
2018-06-08_14:11:01.46640 git.xxxxxx.de @ - - [2018/06/08:16:11:01 +0200] "GET /xxx/xxx/compare/xxx-central%2F0.238-SNAPSHOT...master HTTP/1.1" 502 2916 "https://git.xxxxxx.de/xxx/xxx/compare/master...xxx-central%2F0.238-SNAPSHOT" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0" 60.878
==> /var/log/gitlab/nginx/gitlab_access.log <==
10.2.9.164 - - [08/Jun/2018:16:11:01 +0200] "GET /xxx/xxx/compare/xxx-central%2F0.238-SNAPSHOT...master HTTP/2.0" 502 2916 "https://git.xxxxxx.de/xxx/xxx/compare/master...xxx-central%2F0.238-SNAPSHOT" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0"
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
I, [2018-06-08T16:11:01.759305 #8678] INFO -- : worker=2 ready
perf report of sidekiq shows the following:
12.03% util.rb:23 libruby.so.2.3.0 [.] vm_exec_core
4.21% util.rb:23 libruby.so.2.3.0 [.] gc_sweep_step
3.84% util.rb:23 libruby.so.2.3.0 [.] rb_obj_is_kind_of
2.57% util.rb:23 libjemalloc.so.2 [.] malloc_usable_size
2.50% util.rb:23 libruby.so.2.3.0 [.] ruby_sip_hash24
2.27% util.rb:23 libruby.so.2.3.0 [.] vm_search_method.isra.83
2.23% util.rb:23 libruby.so.2.3.0 [.] ruby_xfree
2.09% util.rb:23 libruby.so.2.3.0 [.] st_lookup
1.90% util.rb:23 libruby.so.2.3.0 [.] vm_call_cfunc
1.53% util.rb:23 libruby.so.2.3.0 [.] str_buf_cat
1.49% util.rb:23 libjemalloc.so.2 [.] arena_dalloc_bin_locked_impl
1.30% util.rb:23 libruby.so.2.3.0 [.] invoke_block_from_c_0
1.19% util.rb:23 libruby.so.2.3.0 [.] rb_enc_cr_str_buf_cat
1.15% util.rb:23 libjemalloc.so.2 [.] free
1.12% util.rb:23 libjemalloc.so.2 [.] malloc
1.04% util.rb:23 libruby.so.2.3.0 [.] objspace_malloc_increase.isra.68
0.97% util.rb:23 libruby.so.2.3.0 [.] rb_enc_set_index
0.93% util.rb:23 [kernel.kallsyms] [k] __lock_text_start
0.93% util.rb:23 libruby.so.2.3.0 [.] rb_enc_from_index
0.93% util.rb:23 libruby.so.2.3.0 [.] rb_enc_get_index
0.93% util.rb:23 libruby.so.2.3.0 [.] rb_wb_protected_newobj_of
0.86% util.rb:23 libruby.so.2.3.0 [.] vm_call0_body.constprop.122
0.78% util.rb:23 libruby.so.2.3.0 [.] vm_exec
0.74% util.rb:23 libruby.so.2.3.0 [.] setup_parameters_complex
0.74% util.rb:23 libruby.so.2.3.0 [.] vm_getivar.isra.102
0.71% util.rb:23 libjemalloc.so.2 [.] je_tcache_bin_flush_small
0.71% util.rb:23 libpthread-2.23.so [.] pthread_mutex_lock
0.71% util.rb:23 libruby.so.2.3.0 [.] method_entry_get
0.71% util.rb:23 libruby.so.2.3.0 [.] rb_any_hash
0.71% util.rb:23 libruby.so.2.3.0 [.] rb_callable_method_entry
0.71% util.rb:23 libruby.so.2.3.0 [.] rb_str_buf_append
0.71% util.rb:23 libruby.so.2.3.0 [.] rb_str_free
0.67% util.rb:23 libruby.so.2.3.0 [.] st_update
0.63% util.rb:23 libruby.so.2.3.0 [.] ary_ensure_room_for_push
0.63% util.rb:23 libruby.so.2.3.0 [.] vm_call_iseq_setup_normal_0start_1params_2locals
0.60% util.rb:23 libruby.so.2.3.0 [.] rb_call0
0.60% util.rb:23 libruby.so.2.3.0 [.] rb_scan_args
0.56% util.rb:23 libruby.so.2.3.0 [.] rb_check_convert_type
0.56% util.rb:23 libruby.so.2.3.0 [.] vm_call_iseq_setup_normal_0start_0params_1locals
0.52% util.rb:23 libc-2.23.so [.] getenv
0.52% util.rb:23 libruby.so.2.3.0 [.] rb_ary_push