Process gitlab named: kthzabor is using 90% of CPU

lately i am having issues with gitlab web interface, i randomly loose access to the web interface.

RAM and Disc usage are below 40%

The only suspicious information is that a process “kthzabor” is using 90% of CPU time.
PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM.
16729 git 20 0 5664640 4,342g 4156 S 1105 37,0 6:05.32 kthzabor

Can someone help me understand or how to find out what does create this process?

Thanks

image948×44 3.72 KB

I suggest you check this post: CVE-2021-22205: How to determine if a self-managed instance has been impacted

Sounds like your gitlab has been compromised. That process is probably crypto mining.

Hi our dedicated gitlab server has the same issue. I found the location of the file
In /dev/shm there is a file kthzabor which was beeing executed.

This is the command I used to check where the file was: find / -name " * kthzabor * "

@rikkert I am having the same issue and by running the top command, was able to find the same kthzabor task at the top. consuming almost 40% of the CPU processing.

were you able to delete the file? was the CPU usage back to normal after that?
Thanks a lot in advance.

another update.
I was able to delete the source file “kthzabor” but then it gets added after a while. Is there any way to find the actual source?
based on the linked question by @iwalker, I tried updating git to the patched version (I had 13.9.0, updated it to 13.9.6). It is still not removing the scripts/ the process kthzabor comes back running after a while.

I haven’t tried to delete the file. I had upgraded the Gitlab to the latest version. Now I haven’t seen that process.

I am monitoring the server closely. I will update here If I face any issues.

I had upgraded the gitlab from 13.0.0 to the latest i.e 14.4.2 Also I had upgraded VM shape as per the gitlab requirements. I am using gitlab CE

You might need to check all cron jobs, for example under /etc/cron.d or even as git user:

crontab -l -u git

there should not be a crontab for git user.

thanks @krishnakvk & @iwalker

I am performing one by one update as I read in the forum that direct upgrade to the latest version breaks gitlab or something. I am also using gitlab CE, I was at 13.4.3 and been upgrading it like this:(13.4.3 > 13.6.0 > 13.7.0 > 13.9.0 > 13.9.6)

Currently at 13.9.6

@iwalker I tried crontab -l -u git but no crontab for git user but there is for root and I think it’s just the default cron jobs right?

Screen Shot 2021-11-15 at 3.59.32 PM1618×600 86.6 KB

Check in /etc/cron.d directory just in case as well as /etc/cron.daily /etc/cron.hourly etc

@iwalker thank you, just checked all of these
cron.d
cron.hourly
cron.daily
cron.weekly

looks nothing suspicious

I had the same problem. kthzabor process had appeared after I deleted kdevtmpfsi (kinsing) mining malware from my server (linux - kdevtmpfsi using the entire CPU - Stack Overflow).
I tried to delete kthzabor file, upgrade Gitlab to 14.4.2, using ClamAV… but it doesn’t go away.
In the end, I had to back up all the data in my Gitlab server and move them to a new instance.
I think maybe it’s a vulnerability issue in old gitlab version.

@iwalker Thanks for your replies, i was able to apply Gitlab upgrades and delete the file “kthzabor” in latest version without having it back in the system.

I confirm that the problem is solved.

@vudinhvinh Thanks for your valuable insight, I am currently having the same issue. No matter how many times I try to delete the “kthzabor” source file and also tried looking into the crons. Still no luck.
I think this was because my gitlab CE version was too old. I am now going to do the same, create a new instance and move gitlab data to the new one.

Is there any official GitLab docs that I could follow for migrating it from the old server to the new one?

First you need to update to version 14.4.2 (lastest version).
(More version in gitlab/gitlab-ce - Packages · packages.gitlab.com)

Then, following this guide to backup and restore: Back up and restore GitLab | GitLab
(Impotant: Back up and restore GitLab | GitLab)

Create a new Gitlab instance (14.4.2) and using created backup file to restore.

@vudinhvinh Thank you!!
Just one question, in order to upgrade to 14.x, do I have to upgrade my server’s OS from ubuntu 16.04 to 18.0 first?

@mike-gitlab yes, as there is not a 14.x package for Xenial. You will need to be careful though, and make sure that the /etc/apt/sources.list.d/gitlab-ce.list file (it will be named slightly differently) does not have bionic in it until you have upgraded the system from 16.04 to 18.04.

For upgrading Gitlab, depending on how old your current version is you need to follow the Gitlab upgrade path - Upgrading GitLab | GitLab the docs have all the commands needed to run the upgrade, or search on the forum as there are plenty of posts that I have written then cover upgrade paths and the steps required.

Hi @mike-gitlab .

I used incron to check if the file was created. If that is the case I delete the file. Next to that I have a cron which checks if the folder has been updated in any way. If that is the case delete the file kthzabor.

@iwalker Thank you! Since there is no 14.x package for Xenial, do you think gitlab will release 14.x for xenial later? Just concerned about upgrading ubuntu.

New Update:
so after updating gitlab-ce to the version “13.12.15” and I restarted the vm, the crypto mining command kthzabor and the files are no longer there. I closely monitored the vm’s CPU usage it’s much much normal now (only using around 10%)

Thank you all for guiding me

Now my concern is whether or not I should update ubuntu 16.04 to 18.0 to be able to run the latest version of gitlab-ce? or can I just run 16.04 for now(thinking gitlab might release 14.x for xenial later) as the 13.12.5 seems to be pretty stable and have removed the mining scripts?

For anyone currently running older version of gitlab-ce and want tho upgrade without breaking, this is the step by step upgrade versions that I did:

13.4.3 > 13.6.0 > 13.7.0 > 13.9.0 > 13.9.6 > 13.12.8 > 13.12.15

sudo apt-get install gitlab-ce=13.4.3-ce.0 //just replace the version

@vudinhvinh Thanks for sharing the article, wow! so this is a well-known exploit affecting many GitLab servers around the world, this makes me wonder if the GitLab enterprise edition is any better in terms of security?

@rikkert, Thanks for sharing incron. that’s something new to me and just googled about it after you mentioned it.

@mike-gitlab no there will not be as they would have already released it otherwise. Xenial went end of life in April 2021 so as soon as this happens other vendors also no longer support it.

You would need to go to 18.04 bionic to go any higher than your current version.