Gitlab 13.9.4 bundle + gitaly cpu hogging

How to Use GitLab
1

Hi I am running a single gitlab instance gitlab ce 13.9.4 on a 24 core 64GB RAM machine and all in a sudden it started showing issues like below

  • Gitlab clone using ssh fails with error ssh_exchange_identification: Connection closed by remote host

  • http based clone and push works fine without any issues

  • We are observing heavy CPU usages on the machine as you can see from below

  • SE linux is disabled on this machine to get rid of any problem related to that

  • Bundle and gitaly are heavy CPU consuming process

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23601 git 20 0 1734880 60712 14428 S 27.8 0.1 35:43.99 gitlab-workhors
23360 git 20 0 1804972 903308 10848 S 24.8 1.4 37:18.08 bundle
23356 git 20 0 1778604 905320 10820 S 23.5 1.4 37:21.28 bundle
23364 git 20 0 1776044 918676 10740 S 22.5 1.4 37:26.65 bundle
23350 git 20 0 1856428 922208 10888 S 19.5 1.4 37:40.93 bundle
23362 git 20 0 1877164 908108 10836 S 19.5 1.4 37:26.75 bundle
23354 git 20 0 1821356 938572 10824 S 18.9 1.4 37:40.56 bundle
23357 git 20 0 1894336 926496 10832 S 18.5 1.4 37:17.50 bundle
23348 git 20 0 1868224 923636 10800 S 15.9 1.4 37:02.91 bundle
23346 git 20 0 2048172 914612 10748 S 13.9 1.4 37:06.15 bundle
23352 git 20 0 1889196 910088 10888 S 13.6 1.4 37:42.31 bundle
23583 git 20 0 2120016 201128 12196 S 10.6 0.3 27:09.89 gitaly
23086 git 20 0 3120228 1.0g 17312 S 6.6 1.6 25:55.65 bundle

Any help here will be appreciated

2

Hi,

Is your installation source-based or did you install it from the Gitlab packages repository, eg: omnibus version. If the result of:

rpm -qa | grep -i gitlab

shows gitlab-ce then you have the omnibus version. In which case, bundle shouldn’t be running on your system. Also, the path for gitaly if this is omnibus install is also wrong, since it would be a path to /opt/gitlab for gitaly.

Sounds like your system has been compromised: CVE-2021-22205: How to determine if a self-managed instance has been impacted

If you have a source installation of gitlab, then bundle would be a process that is present, but I wouldn’t expect it to be running all the time since it should just be for installation. But you will need to check and confirm your installation before we can help further.

Assuming that it is compromised, look at the attached link and you need to be upgrading your server and keeping it regularly upgraded.

1 Like
3

Hi ,
Thanks for your quick response

Its omnibus and is accessible only inside my company network

rpm -qa | grep -i gitlab

gitlab-ce-13.9.4-ce.0.el7.x86_64

4

Actually bundle and bundler do exist, but under /opt/gitlab/embedded/bin for omnibus installations, however that process doesn’t run normally - maybe used during upgrades perhaps. Anyway, definitely sounds like your system is compromised and these are cpuminer processes. Quite a few people have already posted on here about such situations, but generally means killing the processes and upgrading your server to stop it from being compromised.

So you definitely want to be upgrading, and finding out where those files have been downloaded on your system and removing them, checking cron or other potential places that are re-running processes when they are killed. Look to see what is under /tmp as well if the system hasn’t already been rebooted.

1 Like