Process count spikes every 2 minutes

Hi!

In our self-hosted GitLab CE instance (docker-based), we see a spike in the processes from about 230 to about 450 every 2 minutes.

Since there is a bunch of GitLab cron jobs that run every two minutes: Who can help me identity, which one it is?

The following is an excerpt of the TOP command with the additional processes:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
35135 998 20 0 17084 4244 3904 S 0.0 0.1 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35136 998 20 0 7624 3680 3432 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35139 998 20 0 8352 3648 3400 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35140 998 20 0 7624 3632 3380 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35142 998 20 0 8300 3604 3352 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35146 998 20 0 7624 3616 3356 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35148 998 20 0 17072 4380 4036 S 0.0 0.1 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35149 998 20 0 7624 3752 3504 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/ad/3b/ad3b83575249b68aab9602de378314fc221ab07a9b2ab0bb4a245ec649219f45.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options
35150 998 20 0 8352 3708 3460 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch --end-of-options
35151 998 20 0 7624 3668 3416 S 0.0 0.0 0:00.00 /opt/gitlab/embedded/bin/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/4e/47/4e47eb5525df25f94da777993dafa41d9ab2bfa80a89e28f76d42cd46ab082e7.git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input cat-file --batch-check --end-of-options

Hi!

We found the reason for this:
It was not caused by any of the cronjobs, but by our JIRA… :blush:

Hi. We have the same behaviour… We can ask you to tell us more about JIRA integration?

From time to time we have the processes like this:

151601 be/4 git         4.59 M/s    0.00 B/s  0.00 % 55.77 % git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input~d868ca45bbb0f4d04da15eea4fb29c6b95087fe8801bf0a3.git --end-of-options
151598 be/4 git         3.98 M/s    0.00 B/s  0.00 % 55.71 % git -c core.fsyncObjectFiles=true -c gc.auto=0 -c core.autocrlf=input~d868ca45bbb0f4d04da15eea4fb29c6b95087fe8801bf0a3.git --end-of-options

but we do not understand what this process is, what type of jobs…

Hi, the process is GIT itself. In your case, I do not see which Git command it is. In my case, it was git cat-file.
I’d recommend you to observe the NGINX log when the process count spikes again.

As to our solution to this: We’re currently ignoring it :slight_smile:
So any recommendations as to a pragmatic solution are welcome.