Gitaly service utilising more cpu and memory

  • gitaly service utilizing more cpu and memory and creating slowness in gitlab
top - 17:59:30 up 19 days,  8:52,  2 users,  load average: 38.54, 37.99, 45.73
Tasks: 1854 total,  65 running, 1785 sleeping,   4 stopped,   0 zombie
Cpu(s): 54.1%us, 13.8%sy,  0.0%ni, 30.4%id,  0.3%wa,  0.0%hi,  1.3%si,  0.0%st
Mem:  264400568k total, 205368428k used, 59032140k free,  2490156k buffers
Swap: 20971516k total,    37692k used, 20933824k free, 156874104k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
23903 git       20   0 12.4g 287m 6612 S 154.3  0.1  26898:41 gitaly
52520 git       20   0 10.3g 2.4g 186m S 103.5  0.9   5:17.64 bundle
 9142 git       20   0 1686m 805m 310m R 99.3  0.3   0:43.54 git
42098 git       20   0 5449m 878m 162m R 99.3  0.3   2:49.52 bundle
12914 gitlab-r  20   0 5178m 4.6g  548 R 99.0  1.8   0:35.76 redis-server
 6467 git       20   0 1765m 796m 114m R 98.7  0.3   0:36.94 bundle
 6790 git       20   0 1230m 618m  11m R 98.7  0.2   1:56.65 bundle
46494 git       20   0 3298m 647m  44m R 91.0  0.3   1:11.31 bundle
38880 git       20   0 3891m 594m  21m R 77.8  0.2   1:05.78 bundle
53562 git       20   0 3198m 110m 6480 S 73.0  0.0  16815:18 gitlab-workhors
29057 git       20   0 1978m  27m 4248 S 52.7  0.0   0:01.84 gitaly-upload-p
25498 git       20   0 1631m 555m  16m S 42.4  0.2   0:05.44 bundle
29190 git       20   0 1343m 552m 7724 R 40.8  0.2   0:23.62 bundle
36924 git       20   0 1181m 600m  10m S 40.8  0.2   1:49.42 bundle
27580 git       20   0  840m 553m 5716 R 38.6  0.2   0:03.24 bundle
29161 git       20   0  369m  77m  16m S 37.3  0.0   0:01.36 git-upload-pack
33602 gitlab-p  20   0 14.5g 2.0g 2.0g S 35.0  0.8   0:09.52 postgres
16785 git       20   0 2262m 551m 7696 R 32.8  0.2   0:09.52 bundle
45852 git       20   0  987m 560m 7552 S 30.5  0.2   0:19.91 bundle
21483 git       20   0 1383m 558m 8828 S 30.2  0.2   0:28.56 bundle
13312 git       20   0 1505m 544m 7240 R 29.6  0.2   0:10.04 bundle
46932 git       20   0 2762m 580m  16m S 28.0  0.2   0:39.36 bundle
29658 git       20   0 1858m 588m  10m R 27.3  0.2   1:20.86 bundle
13671 git       20   0 3073m 570m  18m R 27.0  0.2   0:48.26 bundle
28396 git       20   0 1811m  25m 4264 S 26.0  0.0   0:02.13 gitaly-upload-p
55267 git       20   0 1934m 596m 8116 S 25.1  0.2   1:00.08 bundle
40538 git       20   0 1070m 559m 8120 S 24.8  0.2   0:35.70 bundle
32672 git       20   0 2492m 618m 9476 R 24.4  0.2   1:06.56 bundle

`

  • Most of the upload-pack process is keep alive in server for long time

Thanks for sharing this info. Could you please open an issue about this in Gitaly issue tracker https://gitlab.com/gitlab-org/gitaly/issues?

Already raised a ticket in another queue : https://gitlab.com/gitlab-org/gitlab-ce/issues/63712