lately i am having issues with gitlab web interface, i randomly loose access to the web interface.
RAM and Disc usage are below 40%
The only suspicious information is that a process “kthzabor” is using 90% of CPU time.
PID UTIL. PR NI VIRT RES SHR S %CPU %MEM TEMPS+ COM.
16729 git 20 0 5664640 4,342g 4156 S 1105 37,0 6:05.32 kthzabor
Can someone help me understand or how to find out what does create this process?
@rikkert I am having the same issue and by running the top command, was able to find the same kthzabor task at the top. consuming almost 40% of the CPU processing.
were you able to delete the file? was the CPU usage back to normal after that?
Thanks a lot in advance.
another update.
I was able to delete the source file “kthzabor” but then it gets added after a while. Is there any way to find the actual source?
based on the linked question by @iwalker, I tried updating git to the patched version (I had 13.9.0, updated it to 13.9.6). It is still not removing the scripts/ the process kthzabor comes back running after a while.
I am performing one by one update as I read in the forum that direct upgrade to the latest version breaks gitlab or something. I am also using gitlab CE, I was at 13.4.3 and been upgrading it like this:(13.4.3 > 13.6.0 > 13.7.0 > 13.9.0 > 13.9.6)
Currently at 13.9.6
@iwalker I tried crontab -l -u git but no crontab for git user but there is for root and I think it’s just the default cron jobs right?
I had the same problem. kthzabor process had appeared after I deleted kdevtmpfsi (kinsing) mining malware from my server (linux - kdevtmpfsi using the entire CPU - Stack Overflow).
I tried to delete kthzabor file, upgrade Gitlab to 14.4.2, using ClamAV… but it doesn’t go away.
In the end, I had to back up all the data in my Gitlab server and move them to a new instance.
I think maybe it’s a vulnerability issue in old gitlab version.
@iwalker Thanks for your replies, i was able to apply Gitlab upgrades and delete the file “kthzabor” in latest version without having it back in the system.
@vudinhvinh Thanks for your valuable insight, I am currently having the same issue. No matter how many times I try to delete the “kthzabor” source file and also tried looking into the crons. Still no luck.
I think this was because my gitlab CE version was too old. I am now going to do the same, create a new instance and move gitlab data to the new one.
Is there any official GitLab docs that I could follow for migrating it from the old server to the new one?
@mike-gitlab yes, as there is not a 14.x package for Xenial. You will need to be careful though, and make sure that the /etc/apt/sources.list.d/gitlab-ce.list file (it will be named slightly differently) does not have bionic in it until you have upgraded the system from 16.04 to 18.04.
For upgrading Gitlab, depending on how old your current version is you need to follow the Gitlab upgrade path - Upgrading GitLab | GitLab the docs have all the commands needed to run the upgrade, or search on the forum as there are plenty of posts that I have written then cover upgrade paths and the steps required.
I used incron to check if the file was created. If that is the case I delete the file. Next to that I have a cron which checks if the folder has been updated in any way. If that is the case delete the file kthzabor.
@iwalker Thank you! Since there is no 14.x package for Xenial, do you think gitlab will release 14.x for xenial later? Just concerned about upgrading ubuntu.
New Update:
so after updating gitlab-ce to the version “13.12.15” and I restarted the vm, the crypto mining command kthzabor and the files are no longer there. I closely monitored the vm’s CPU usage it’s much much normal now (only using around 10%)
Thank you all for guiding me
Now my concern is whether or not I should update ubuntu 16.04 to 18.0 to be able to run the latest version of gitlab-ce? or can I just run 16.04 for now(thinking gitlab might release 14.x for xenial later) as the 13.12.5 seems to be pretty stable and have removed the mining scripts?
For anyone currently running older version of gitlab-ce and want tho upgrade without breaking, this is the step by step upgrade versions that I did:
sudo apt-get install gitlab-ce=13.4.3-ce.0 //just replace the version
@vudinhvinh Thanks for sharing the article, wow! so this is a well-known exploit affecting many GitLab servers around the world, this makes me wonder if the GitLab enterprise edition is any better in terms of security?
@rikkert, Thanks for sharing incron. that’s something new to me and just googled about it after you mentioned it.
@mike-gitlab no there will not be as they would have already released it otherwise. Xenial went end of life in April 2021 so as soon as this happens other vendors also no longer support it.
You would need to go to 18.04 bionic to go any higher than your current version.
