Git Processes Killing CPU

General
1

The other day my self-hosted instance started experiencing resource issues. Taking a look from htop, all 8 cpus were maxed out, with the highest consumers being several processes started from “git”, the command being a catalina bootstrap start.

After a few days of observation and trying several things, I noticed a few patterns. These processes kick on a few seconds after I noticed a few gitlab-rails processes start. Both the bootstrap processes and gitlab-rails processes would kick on after a few minutes of restarting the server even with runsvdir disabled and gitlab-ctl stopped. They also appear again after killing all “git” processes. The problem goes away if I rename the user to something else and specify the gid and uid and reconfigure.

Do I have a virus or some other kind of exploitation, or are there settings I can tune for rails and whatever this mystery process is and what indeed is it?

I am running gitlab-ce-13.1.14.

Also to note, disabling puma, workhorse, or prometheus and reconfiguring has no impact on this problem.

UPDATE:
After leaving it alone for a half hour, I came back to find the issue still occurs with the new username and uid and gid. Attached are screenshots of the processes that would fit. Note that “gituser” is the rename of the old “git” user.

Mystery-Processes
Mystery-Processes1475×1129 422 KB

2

Looks like your server was exploited you need to upgrade it.

Plenty of others have posted here recently check this post Gitlab , git juma - #2 by iwalker

1 Like
3

Thank you for your quick response!

I looked at CVE-2021-22205: How to determine if a self-managed instance has been impacted - #17 by mmcintosh but I did not see anything regarding the ExifTool in my logs. I do however see null user_id and username entries for the /users/sign_in POST command in the production_json.log.

Does this still sound like we have been exploited by this?

4

@iwalker and the ip address is not one I recognize or expect.

5

Well, you have processes eating your CPU, so yes whilst you may not find such entries in your log it has been compromised.

Also tomcat is not part of gitlab, so unless you installed this yourself, then chances are this is a rogue process on your machine and needs to be removed.

6

Yeah I have already begun the process of backing up our data and moving to a new machine. Thank you for confirming our system being compromised!

1 Like
7

Remember to get the /etc/gitlab/gitlab.rb file as well as gitlab-secrets.json in this directory as well, as both are required. Under /var/opt/gitlab/backups you should be able to find the latest backup file of which you can also copy to your new server. Then you can restore the system. The process is more or less:

  1. Install exact same version of Gitlab on the new server which is installed on the old server (do not run reconfigure yet!).
  2. Copy gitlab.rb and gitlab-secrets.json to /etc/gitlab.
  3. Now run: gitlab-ctl reconfigure to get your system up and running clean.
  4. Place backup file in /var/opt/gitlab/backups and set appropriate permissions (gitlab docs for backup/restore will show this).
  5. Start gitlab restore as per gitlab restore docs.
1 Like