Experiments and Facts
My system has 40 cores, 128GB RAM and a 240GB SSD for OS and applications, on which the lastest (v15.3) GitLab CE docker instance is installed. The GitLab instance has 81 projects (<3GB), 63 users and 5 groups.
The instance is configured to run 20 puma processes with at most 4 threads per process. Then, I choose a project that has 58 commits, 3 branches and 175.5MB project storage, and access the project page as a performance test. For a cold access, the performance bar reads
697ms/103 pg,0 gitaly,29ms/78 redis,0 es,0 external,253 MB memory,3410|3483|3604/45 total
and production.log reads
Started GET "/robomaster/auto-aim" for 172.17.0.1 at 2022-09-01 14:56:59 +0000
Processing by ProjectsController#show as HTML
Parameters: {"namespace_id"=>"robomaster", "id"=>"auto-aim"}
Rendered layout layouts/project.html.haml (Duration: 2446.9ms | Allocations: 1358866)
Completed 200 OK in 2840ms (Views: 1902.7ms | ActiveRecord: 692.1ms | Elasticsearch: 0.0ms | Allocations: 1492279)
Using ab(ApacheBench) to access the page for 10000 times with 16 concurrent threads gives
Server Software: nginx/1.21.6
Server Hostname: [REDACTED]
Server Port: 80
Document Path: /robomaster/auto-aim
Document Length: 130198 bytes
Concurrency Level: 16
Time taken for tests: 380.812 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 1308872490 bytes
HTML transferred: 1301980000 bytes
Requests per second: 26.26 [#/sec] (mean)
Time per request: 609.298 [ms] (mean)
Time per request: 38.081 [ms] (mean, across all concurrent requests)
Transfer rate: 3356.50 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 1
Processing: 485 604 120.3 587 3623
Waiting: 484 604 120.2 587 3623
Total: 485 605 120.3 588 3624
Percentage of the requests served within a certain time (ms)
50% 588
66% 606
75% 619
80% 629
90% 657
95% 686
98% 900
99% 1079
100% 3624 (longest request)
and statistics from production.log are
Total Views ActiveRecord Allocations
count 10000.000000 10000.00000 10000.000000 1.000000e+04
mean 546.790000 462.09148 41.852460 2.121500e+05
std 106.499338 74.91495 32.429246 3.978384e+04
min 445.000000 374.10000 21.000000 2.081160e+05
25% 508.000000 429.80000 34.300000 2.082610e+05
50% 530.000000 448.60000 39.800000 2.082810e+05
75% 559.000000 476.10000 41.800000 2.083110e+05
max 2887.000000 2018.00000 646.300000 1.291930e+06
Questions
- Considering the load against the hardware, the performance is way below my expectation. Or it could just be that I expect too much from GitLab. Does these data indicate real performance issues?
- If these are performance issues, which aspects should I investigate further? Is there any tuning I could try out?