Gitlab 501 error - cpu 100%usage

I am using gitlab “gitlab-ce-13.1.4-ce.0.el6.x86_64” on centos.

I am facing 501 error and not able to access the gitlab ui. after proceeded gitlabl-ctl restart and reconfigure, it is not working. CPU goes 100% usages.

Gitlab is working fine, after rebooted the server. but it working till next 20 to 45 minutes only and facing same 501 error.

I observed that cpu is less than 5% only after reboot. when a git cron job is trigger at the time, below two process is created and gitlab-redis process consumed more than 80% cpu.

bash -s /opt/gitlab/

Can you please provide the guidelines to fix the issue.

I am not sure why creating issue after running gitlab cron jobs.

LOG: /var/log/cron
Dec 13 16:14:57 TEST crontab[4179]: (git) AUTH (crontab command not allowed)
Dec 13 16:14:57 TEST crontab[4181]: (git) AUTH (crontab command not allowed)

Is that might be cause of project repository storage size, number of respository limit

Hi, see this thread: Gitlab-redis process CPU spike

Most likely cron as git user is starting crypto miner processes which is killing your server.

Yes, I got it. After stopped the below service, gitlab started to working again. Also i referred the given thread. Also verified and deleted unwanted cron from git user cron and /etc/cron.d

But those service is started after few minutes. Is that possible to fix in permenantly.
I am using gitlab “gitlab-ce-13.1.4-ce.0.el6.x86_64” on centos 6.

bash -s /opt/gitlab/

Thank you for guidelines.

You need to check cron, as this is most likely what is running the processes. Check under /etc/cron.d to see if any files have been placed here. But rather, it will be cron under the git user so:

crontab -l -u git

should give results if something is being run every X minutes or something. You can then edit that cron with:

crontab -e -u git

and then remove the lines that are running the processes. There shouldn’t be a cron for git anyway, so it should be safe to remove anything on that crontab for the git user.

Hi iwalker,

Thank you. I checked the git user crontab and /etc/cron.d also; No cronjobs for git users.

But when i kill those two affected processes, it will start in less than 30 minutes. I got the below cron job logs when those processes started.

Dec 14 08:46:53 TEST crontab[28879]: (git) AUTH (crontab command not allowed)
Dec 14 08:46:53 TEST crontab[28881]: (git) AUTH (crontab command not allowed)

I can see the lot of cron jobs under my gitlab below url.

Is that possible to identify the cron job in ui which is possible to affect it.

Also can you please share where we can find the gitlab cron logs. so that i can able to find which cron job is triggering the suspicious process and will disable it

Once again thank you for your support.

Well, you could share a screenshot of the cron jobs in the gui so we could attempt to find it. Alternatively, depending on the distribution you are installed on, but assuming Debian/Ubuntu, then look at /var/log/syslog and check this:

cat /var/log/syslog | grep -i cron

if RHEL based, then /var/log/messages should show, or even /var/log/cron would be filtered better. These files can hint if the system cron is being used.

Look at /var/log/gitlab/sidekiq/current as a log file also.

I am using centos “RHEL” and getting log “/var/log/cron” as below. From the log, cant able to find which process/job is executed

Dec 14 08:46:53 TEST crontab[28879]: (git) AUTH (crontab command not allowed)
Dec 14 08:46:53 TEST crontab[28881]: (git) AUTH (crontab command not allowed)

Please see the screenshot.
Thank you.

I don’t see anything in Gitlab cron suspicious, so if anything is being ran from cron, it can only be from the system one now.

Try doing this:

su git
crontab -l

see if it gives results.

Ok thanks. when those suspected two process is running, i cant able to login as git user using su git. after killed those processes only, able to logged in.

su - git


su git

sh-4.1$ crontab -l
You (git) are not allowed to use this program (crontab)
See crontab(1) for more information

I am suspecting this process. any idea about this.
ps axu | grep ‘bash -s /opt/gitlab’
git 7384 1.0 0.0 11344 1288 ? S 10:28 0:07 bash -s /opt/gitlab/

Thank you.

OK, if it’s not letting you view it, check under /var/spool/cron and see if a file exists here for git. If so then you can view that file using cat, eg:

cat /var/spool/cron/git

assuming that the file is called git. I have differently under Debian, /var/spool/cron/crontabs/ and a file would exist for the git user here for example. So you might find yours similar like my example above. If the /var/spool/cron/git file exists, then you can just delete it.

The bash process doesn’t exist on my system, so this is obviously part of the problem.

Either way, once you’ve managed to stop the processes from starting, you need to upgrade your Gitlab.

As for the cron error, maybe git is blocked in /etc/cron.deny or you need to add it to /etc/cron.allow. But using the commands to get to /var/spool/cron as root will get around it and allow is to delete it without editing those files.

Yes, this file “/var/spool/cron/git” is available and it is seems to be empty; Now I deleted this file.

Also i upgraded the gitlab ce from 13.1.4 into 13.6.7.

I will reboot the server once and verify it.

Thank you for your valuable support

After rebooted, those process is started again;

after killed those process and stopped the “gitlab-ctl stop”, then it is not starting and server cpu is normal.

I am not sure how to stop the suspected process in permanently.

Please share your thoughts to finding the issue.

Also I can find the gitlab-redis process using pstree

pstree 1


You need to stop those. From the post I linked before, the open bash process is constantly downloading something from the internet which is overriding everything you are trying to achieve.

I never had this infection, but you will have to search your server and find everything related to this and stop it permanently. If not cron, then it is picking it up somewhere else.

It is stopped and added one bash script to check and stop those 2 process for timebeing.

I am looking the permenant solution.

Thank you.

Just out of interest, check in /opt/gitlab if there are any .bashrc files or something. I’m curious as to why bash -s /opt/gitlab so I have a feeling maybe it’s reading a .bashrc file in /opt/gitlab.


ls -lha /opt/gitlab

and see what it lists. Then we can work out if something is there that shouldn’t be.

Checked and verified no hidden files and .bashrc on below path.
ls -lha /opt/gitlab

I found that suspected those process is started in every 19 to 20 minutes interval exactly. This might be useful to further digging.

Please see the screenshot.

Thank you for support.