Hi,
I’m running Gitlab-CE 15.8.1 on TrueNAS scale, meaning it is a docker running inside kubernetes k3s.
This is a “to play with” gitlab, meaning I’m mostly alone on it with very few projects and nothing complex setup (no real CD/CI,runners yet).
In that context, I would be expecting the CPU usage to be near 0 most of time. But I observed (first in TrueNAS GUI, then in the gitlab-ce instance), that the CPU is constantly going from a low value to 20-30% CPU, back and forth, constantly. The culprit is bundle run by user git (see below).
Before upgrading to 15.8.1, this process had significantly more TIME+ (seen in top cmd) than other processes.
Is any of this normal ? Do I have unwanted software in my instance (I’ve seens a forum post talking about that, but for versions of gitlab way older).
I’ve added a few random things from fast-stat if it can help.
root@gitlab-ce:/var/log/gitlab# ./fast-stats top gitlab-rails/production_json.log
Totals
COUNT: 4669
RPS: 0.07
DUR: 7m34.2s
DB: 2.5s
REDIS: 5.8s
GITALY: 0.1s
QUEUE: 1.6s
CPU: 8m09.8s
MEM: 6.87 GiB
Top 10 Path by Duration
PATH COUNT RPS DUR DB REDIS GITALY QUEUE CPU MEM FAIL_CT
/-/metrics 4621 / 98.97% 0.07 / 98.97% 7m05.6s / 93.70% 0.0s / 0.10% 5.4s / 92.42% 0.0s / 0.00% 0.0s / 0.00% 7m40.7s / 94.05% 5.35 GiB / 77.95% 0 / 0.00%
/help/update/index 5 / 0.11% 0.00 / 0.11% 7.1s / 1.57% 0.0s / 1.09% 0.0s / 0.85% 0.0s / 0.00% 0.1s / 6.85% 7.1s / 1.45% 733.66 MiB / 10.43% 0 / 0.00%
/ 8 / 0.17% 0.00 / 0.17% 6.0s / 1.32% 0.9s / 34.91% 0.1s / 2.32% 0.1s / 67.30% 0.4s / 27.26% 6.2s / 1.27% 271.84 MiB / 3.87% 0 / 0.00%
/gitlab-instance-984ecfa7/monitoring 2 / 0.04% 0.00 / 0.04% 4.0s / 0.88% 0.3s / 12.58% 0.0s / 0.78% 0.0s / 32.70% 0.1s / 3.23% 4.2s / 0.86% 161.35 MiB / 2.29% 0 / 0.00%
/users/sign_in 5 / 0.11% 0.00 / 0.11% 2.8s / 0.61% 0.5s / 18.55% 0.0s / 0.84% 0.0s / 0.00% 0.1s / 6.28% 2.4s / 0.49% 102.73 MiB / 1.46% 0 / 0.00%
/help 2 / 0.04% 0.00 / 0.04% 2.1s / 0.46% 0.0s / 0.96% 0.0s / 0.41% 0.0s / 0.00% 0.1s / 3.77% 2.1s / 0.43% 74.48 MiB / 1.06% 0 / 0.00%
/dashboard/todos 3 / 0.06% 0.00 / 0.06% 2.0s / 0.44% 0.0s / 1.19% 0.1s / 0.97% 0.0s / 0.00% 0.1s / 4.72% 2.3s / 0.48% 50.69 MiB / 0.72% 0 / 0.00%
/dashboard/projects.atom 3 / 0.06% 0.00 / 0.06% 2.0s / 0.43% 0.6s / 24.56% 0.0s / 0.13% 0.0s / 0.00% 0.1s / 6.06% 1.4s / 0.29% 52.48 MiB / 0.75% 0 / 0.00%
/admin/application_settings/general 1 / 0.02% 0.00 / 0.02% 1.2s / 0.25% 0.0s / 0.75% 0.0s / 0.21% 0.0s / 0.00% 0.0s / 2.70% 1.1s / 0.23% 36.43 MiB / 0.52% 0 / 0.00%
/admin 1 / 0.02% 0.00 / 0.02% 0.7s / 0.14% 0.1s / 2.29% 0.0s / 0.32% 0.0s / 0.00% 0.0s / 0.67% 0.6s / 0.13% 24.36 MiB / 0.35% 0 / 0.00%
Top 10 Project by Duration
PROJECT COUNT RPS DUR DB REDIS GITALY QUEUE CPU MEM FAIL_CT
gitlab-instance-984ecfa7/monitoring 2 / 0.04% 0.00 / 0.04% 4.0s / 0.88% 0.3s / 12.58% 0.0s / 0.78% 0.0s / 32.70% 0.1s / 3.23% 4.2s / 0.86% 161.35 MiB / 2.29% 0 / 0.00%
Top 10 User by Duration
USER COUNT RPS DUR DB REDIS GITALY QUEUE CPU MEM FAIL_CT
root 23 / 0.49% 0.00 / 0.49% 26.3s / 5.80% 2.3s / 91.00% 0.4s / 6.23% 0.1s / 100.00% 0.8s / 49.63% 26.4s / 5.39% 1.43 GiB / 20.81% 0 / 0.00%
root@gitlab-ce:/var/log/gitlab# ./fast-stats top gitaly/current
Totals
COUNT: 4
RPS: 0.00
DUR: 0.0s
CPU: 0.0s
GIT_RSS: 574.22 MiB
DISK_R: 30
Top 10 Project by Duration
PROJECT COUNT RPS DUR CPU GIT_RSS DISK_R FAIL_CT
gitlab-instance-984ecfa7/monitoring 3 / 75.00% 0.00 / 75.00% 0.0s / 97.88% 0.0s / 100.00% 574.22 KiB / 100.00% 30 / 100.00% 0 / 0.00%
Top 10 User by Duration
USER COUNT RPS DUR CPU GIT_RSS DISK_R FAIL_CT
root 3 / 75.00% 0.00 / 75.00% 0.0s / 97.88% 0.0s / 100.00% 574.22 KiB / 100.00% 30 / 100.00% 0 / 0.00%
Top 10 Client by Duration
CLIENT COUNT RPS DUR CPU GIT_RSS DISK_R FAIL_CT
gitlab-web 3 / 75.00% 0.00 / 75.00% 0.0s / 97.88% 0.0s / 100.00% 574.22 KiB / 100.00% 30 / 100.00% 0 / 0.00%
gitlab-sidekiq 1 / 25.00% 0.00 / 25.00% 0.0s / 2.12% 0.0s / 0.00% 0 B / 0.00% 0 / 0.00% 0 / 0.00%