Web application very slow - unicorn processes?

How to Use GitLab
1

Hey all,

I have a self-hosted GitLab running. From time to time, we got a few hundred people - which is not that extraordinary - working on GitLab web interface at the same time. It seems the GitLab web interface is very slow at these times (like in: we’re getting timeouts, or have to wait in the order of minutes). I’ve already increased the unicorn workers/processes to 16. Things see, to be a bit better, but still, I’m not sure to what extends increasing these workers/processes will scale properly.

The server is not overloaded (runs on about ~50% on RAM, ~60-80% on CPU). Network also seems ok, since ssh works fine, and git pull and git push etc. also works ok - not blazing fast, but workable.

What can be the reason for the slow web application?
Are there any recommendations for unicorn workers?

Thanks a lot,
Kurt

2

I’m getting a similar issue with prolonged page load times. Sometimes it’s ~200-300ms, other times it’s ~3-6secs, and it’s not consistent. Unicorn processes = 13. Probably an issue w/ gitlab-workhorse or unicorn.

Gitlab-Workhorse Logs

TIME: 219ms

2019-03-27_23:12:03.69136 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET / HTTP/1.1" 200 87925 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.219

2019-03-27_23:12:03.73739 time="2019-03-27T19:12:03-04:00" level=info msg="Send static file" correlation_id=MGx0bJI5kj8 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/print-c8ff536271f8974b8a9a5f75c0ca25d2b8c1dceb4cff3c01d1603862a0bdcbfc.css method=GET uri=/assets/print-c8ff536271f8974b8a9a5f75c0ca25d2b8c1dceb4cff3c01d1603862a0bdcbfc.css

2019-03-27_23:12:03.73743 time="2019-03-27T19:12:03-04:00" level=info msg="Send static file" correlation_id=dexElj7TE73 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/highlight/themes/white-a165d47ce52cf24c29686366976ae691bd9addb9641a6abeb3ba6d1823b89aa8.css method=GET uri=/assets/highlight/themes/white-a165d47ce52cf24c29686366976ae691bd9addb9641a6abeb3ba6d1823b89aa8.css

2019-03-27_23:12:03.73751 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET /assets/print-c8ff536271f8974b8a9a5f75c0ca25d2b8c1dceb4cff3c01d1603862a0bdcbfc.css HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:12:03.73761 time="2019-03-27T19:12:03-04:00" level=info msg="Send static file" correlation_id=XvIcr8dOtVa encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/application-f7d1309f3caef67cb63bd114c85e73b323a97d145ceca7d6ef3c1c010078c649.css method=GET uri=/assets/application-f7d1309f3caef67cb63bd114c85e73b323a97d145ceca7d6ef3c1c010078c649.css

2019-03-27_23:12:03.73763 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET /assets/highlight/themes/white-a165d47ce52cf24c29686366976ae691bd9addb9641a6abeb3ba6d1823b89aa8.css HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:12:03.73768 time="2019-03-27T19:12:03-04:00" level=info msg="Send static file" correlation_id=rEJWnrrT9o2 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/runtime.7ea97dc0.bundle.js method=GET uri=/assets/webpack/runtime.7ea97dc0.bundle.js

2019-03-27_23:12:03.73772 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET /assets/application-f7d1309f3caef67cb63bd114c85e73b323a97d145ceca7d6ef3c1c010078c649.css HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.001

2019-03-27_23:12:03.73783 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET /assets/webpack/runtime.7ea97dc0.bundle.js HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:12:03.73805 time="2019-03-27T19:12:03-04:00" level=info msg="Send static file" correlation_id=A5NkgLsXYV6 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/main.74b09fa9.chunk.js method=GET uri=/assets/webpack/main.74b09fa9.chunk.js

2019-03-27_23:12:03.73816 time="2019-03-27T19:12:03-04:00" level=info msg="Send static file" correlation_id=3W3AYairzca encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/pages.root.e87f1e8c.chunk.js method=GET uri=/assets/webpack/pages.root.e87f1e8c.chunk.js

2019-03-27_23:12:03.73824 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET /assets/webpack/main.74b09fa9.chunk.js HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:12:03.73832 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET /assets/webpack/pages.root.e87f1e8c.chunk.js HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:12:03.81974 time="2019-03-27T19:12:03-04:00" level=info msg="Send static file" correlation_id=LwOdqdPJFe8 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/icons-24aaa921aa9e411162e6913688816c79861d0de4bee876cf6fc4c794be34ee91.svg method=GET uri=/assets/icons-24aaa921aa9e411162e6913688816c79861d0de4bee876cf6fc4c794be34ee91.svg

2019-03-27_23:12:03.81986 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:03 -0400] "GET /assets/icons-24aaa921aa9e411162e6913688816c79861d0de4bee876cf6fc4c794be34ee91.svg HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:12:04.14043 time="2019-03-27T19:12:04-04:00" level=info msg="Send file" correlation_id=byw0iUGImia file=/opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/user/avatar/2/avatar.png method=GET uri="/uploads/-/system/user/avatar/2/avatar.png?width=23"

2019-03-27_23:12:04.14102 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /uploads/-/system/user/avatar/2/avatar.png?width=23 HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.035

2019-03-27_23:12:04.19305 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /apps/games/minecraft/sevtech-docker/avatar HTTP/1.1" 200 341036 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.087

2019-03-27_23:12:04.19564 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /apps/odoo-docker/avatar HTTP/1.1" 200 17647 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.082

2019-03-27_23:12:04.19565 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /core/nginx-docker/avatar HTTP/1.1" 200 16437 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.090

2019-03-27_23:12:04.19566 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /apps/plex-docker/avatar HTTP/1.1" 200 175248 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.089

2019-03-27_23:12:04.20287 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /apps/games/spacestationgalaxy/SpaceStationGalaxy/avatar HTTP/1.1" 200 118642 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.092

2019-03-27_23:12:04.21139 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /apps/speedtest-docker/avatar HTTP/1.1" 200 125488 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.103

2019-03-27_23:12:04.21309 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /apps/portainer-docker/avatar HTTP/1.1" 200 4253 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.087

2019-03-27_23:12:04.22343 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:12:04 -0400] "GET /apps/games/minecraft/atm3-docker/avatar HTTP/1.1" 200 325083 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.098

TIME: 2924ms

2019-03-27_23:11:11.62610 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET / HTTP/1.1" 200 87925 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 3.152

2019-03-27_23:11:11.65558 time="2019-03-27T19:11:11-04:00" level=info msg="Send static file" correlation_id=6XSQxRMA6Z4 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/print-c8ff536271f8974b8a9a5f75c0ca25d2b8c1dceb4cff3c01d1603862a0bdcbfc.css method=GET uri=/assets/print-c8ff536271f8974b8a9a5f75c0ca25d2b8c1dceb4cff3c01d1603862a0bdcbfc.css

2019-03-27_23:11:11.65561 time="2019-03-27T19:11:11-04:00" level=info msg="Send static file" correlation_id=MzArPbVleBa encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/pages.root.e87f1e8c.chunk.js method=GET uri=/assets/webpack/pages.root.e87f1e8c.chunk.js

2019-03-27_23:11:11.65562 time="2019-03-27T19:11:11-04:00" level=info msg="Send static file" correlation_id=pEUXzJstoU6 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/highlight/themes/white-a165d47ce52cf24c29686366976ae691bd9addb9641a6abeb3ba6d1823b89aa8.css method=GET uri=/assets/highlight/themes/white-a165d47ce52cf24c29686366976ae691bd9addb9641a6abeb3ba6d1823b89aa8.css

2019-03-27_23:11:11.65563 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET /assets/print-c8ff536271f8974b8a9a5f75c0ca25d2b8c1dceb4cff3c01d1603862a0bdcbfc.css HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.001

2019-03-27_23:11:11.65565 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET /assets/highlight/themes/white-a165d47ce52cf24c29686366976ae691bd9addb9641a6abeb3ba6d1823b89aa8.css HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:11:11.65567 time="2019-03-27T19:11:11-04:00" level=info msg="Send static file" correlation_id=NshGVehQOy7 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/application-f7d1309f3caef67cb63bd114c85e73b323a97d145ceca7d6ef3c1c010078c649.css method=GET uri=/assets/application-f7d1309f3caef67cb63bd114c85e73b323a97d145ceca7d6ef3c1c010078c649.css

2019-03-27_23:11:11.65568 time="2019-03-27T19:11:11-04:00" level=info msg="Send static file" correlation_id=b0szV4JHUk2 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/runtime.7ea97dc0.bundle.js method=GET uri=/assets/webpack/runtime.7ea97dc0.bundle.js

2019-03-27_23:11:11.65570 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET /assets/webpack/pages.root.e87f1e8c.chunk.js HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:11:11.65584 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET /assets/application-f7d1309f3caef67cb63bd114c85e73b323a97d145ceca7d6ef3c1c010078c649.css HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.001

2019-03-27_23:11:11.65611 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET /assets/webpack/runtime.7ea97dc0.bundle.js HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:11:11.65616 time="2019-03-27T19:11:11-04:00" level=info msg="Send static file" correlation_id=FO2gdPvHQC2 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/main.74b09fa9.chunk.js method=GET uri=/assets/webpack/main.74b09fa9.chunk.js

2019-03-27_23:11:11.65627 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET /assets/webpack/main.74b09fa9.chunk.js HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:11:11.79198 time="2019-03-27T19:11:11-04:00" level=info msg="Send static file" correlation_id=XAOyvfcNr13 encoding=gzip file=/opt/gitlab/embedded/service/gitlab-rails/public/assets/icons-24aaa921aa9e411162e6913688816c79861d0de4bee876cf6fc4c794be34ee91.svg method=GET uri=/assets/icons-24aaa921aa9e411162e6913688816c79861d0de4bee876cf6fc4c794be34ee91.svg

2019-03-27_23:11:11.79208 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:11 -0400] "GET /assets/icons-24aaa921aa9e411162e6913688816c79861d0de4bee876cf6fc4c794be34ee91.svg HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.000

2019-03-27_23:11:12.04791 time="2019-03-27T19:11:12-04:00" level=info msg="Send file" correlation_id=oVuhYhyk0N9 file=/opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/user/avatar/2/avatar.png method=GET uri="/uploads/-/system/user/avatar/2/avatar.png?width=23"

2019-03-27_23:11:12.04808 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /uploads/-/system/user/avatar/2/avatar.png?width=23 HTTP/1.1" 304 0 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.046

2019-03-27_23:11:12.09776 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /apps/speedtest-docker/avatar HTTP/1.1" 200 125488 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.095

2019-03-27_23:11:12.09792 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /core/nginx-docker/avatar HTTP/1.1" 200 16437 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.096

2019-03-27_23:11:12.10818 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /apps/games/spacestationgalaxy/SpaceStationGalaxy/avatar HTTP/1.1" 200 118642 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.100

2019-03-27_23:11:12.10885 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /apps/portainer-docker/avatar HTTP/1.1" 200 4253 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.101

2019-03-27_23:11:12.10919 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /apps/plex-docker/avatar HTTP/1.1" 200 175248 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.107

2019-03-27_23:11:12.10932 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /apps/games/minecraft/sevtech-docker/avatar HTTP/1.1" 200 341036 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.107

2019-03-27_23:11:12.11009 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /apps/games/minecraft/atm3-docker/avatar HTTP/1.1" 200 325083 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.102

2019-03-27_23:11:12.11143 git.REDACTED.com 127.0.0.1:0 - - [2019/03/27:19:11:12 -0400] "GET /apps/odoo-docker/avatar HTTP/1.1" 200 17647 "https://git.REDACTED.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.104