Process count spikes every 2 minutes

Hi!

In our self-hosted GitLab CE instance (docker-based), we see a spike in the processes from about 230 to about 450 every 2 minutes.

Since there is a bunch of GitLab cron jobs that run every two minutes: Who can help me identity, which one it is?

The following is an excerpt of the TOP command with the additional processes:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
35135 998 20 0 17084 4244 3904 S 0.0 0.1 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35136 998 20 0 7624 3680 3432 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35139 998 20 0 8352 3648 3400 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35140 998 20 0 7624 3632 3380 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35142 998 20 0 8300 3604 3352 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35146 998 20 0 7624 3616 3356 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35148 998 20 0 17072 4380 4036 S 0.0 0.1 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35149 998 20 0 7624 3752 3504 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35150 998 20 0 8352 3708 3460 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35151 998 20 0 7624 3668 3416 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options

Hi!

We found the reason for this:
It was not caused by any of the cronjobs, but by our JIRA…

Hi. We have the same behaviour… We can ask you to tell us more about JIRA integration?

From time to time we have the processes like this:

151601 be/4 git         4.59 M/s    0.00 B/s  0.00 % 55.77 % git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input~d868ca45bbb0f4d04da15eea4fb29c6b95087fe8801bf0a3.git --end-of-options
151598 be/4 git         3.98 M/s    0.00 B/s  0.00 % 55.71 % git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input~d868ca45bbb0f4d04da15eea4fb29c6b95087fe8801bf0a3.git --end-of-options

but we do not understand what this process is, what type of jobs…

Hi, the process is GIT itself. In your case, I do not see which Git command it is. In my case, it was git cat-file.
I’d recommend you to observe the NGINX log when the process count spikes again.

As to our solution to this: We’re currently ignoring it
So any recommendations as to a pragmatic solution are welcome.