I am using gitlab “gitlab-ce-13.1.4-ce.0.el6.x86_64” on centos.
I am facing 501 error and not able to access the gitlab ui. after proceeded gitlabl-ctl restart and reconfigure, it is not working. CPU goes 100% usages.
Gitlab is working fine, after rebooted the server. but it working till next 20 to 45 minutes only and facing same 501 error.
I observed that cpu is less than 5% only after reboot. when a git cron job is trigger at the time, below two process is created and gitlab-redis process consumed more than 80% cpu.
gitlab-redis
bash -s /opt/gitlab/
Can you please provide the guidelines to fix the issue.
I am not sure why creating issue after running gitlab cron jobs.
LOG: /var/log/cron
Dec 13 16:14:57 TEST crontab[4179]: (git) AUTH (crontab command not allowed)
Dec 13 16:14:57 TEST crontab[4181]: (git) AUTH (crontab command not allowed)
Is that might be cause of project repository storage size, number of respository limit
Yes, I got it. After stopped the below service, gitlab started to working again. Also i referred the given thread. Also verified and deleted unwanted cron from git user cron and /etc/cron.d
But those service is started after few minutes. Is that possible to fix in permenantly.
I am using gitlab “gitlab-ce-13.1.4-ce.0.el6.x86_64” on centos 6.
You need to check cron, as this is most likely what is running the processes. Check under /etc/cron.d to see if any files have been placed here. But rather, it will be cron under the git user so:
crontab -l -u git
should give results if something is being run every X minutes or something. You can then edit that cron with:
crontab -e -u git
and then remove the lines that are running the processes. There shouldn’t be a cron for git anyway, so it should be safe to remove anything on that crontab for the git user.
Thank you. I checked the git user crontab and /etc/cron.d also; No cronjobs for git users.
But when i kill those two affected processes, it will start in less than 30 minutes. I got the below cron job logs when those processes started.
Dec 14 08:46:53 TEST crontab[28879]: (git) AUTH (crontab command not allowed)
Dec 14 08:46:53 TEST crontab[28881]: (git) AUTH (crontab command not allowed)
Is that possible to identify the cron job in ui which is possible to affect it.
Also can you please share where we can find the gitlab cron logs. so that i can able to find which cron job is triggering the suspicious process and will disable it
Well, you could share a screenshot of the cron jobs in the gui so we could attempt to find it. Alternatively, depending on the distribution you are installed on, but assuming Debian/Ubuntu, then look at /var/log/syslog and check this:
cat /var/log/syslog | grep -i cron
if RHEL based, then /var/log/messages should show, or even /var/log/cron would be filtered better. These files can hint if the system cron is being used.
Look at /var/log/gitlab/sidekiq/current as a log file also.
I am using centos “RHEL” and getting log “/var/log/cron” as below. From the log, cant able to find which process/job is executed
Dec 14 08:46:53 TEST crontab[28879]: (git) AUTH (crontab command not allowed)
Dec 14 08:46:53 TEST crontab[28881]: (git) AUTH (crontab command not allowed)
Ok thanks. when those suspected two process is running, i cant able to login as git user using su git. after killed those processes only, able to logged in.
su - git
Killed
su git
sh-4.1$ crontab -l
You (git) are not allowed to use this program (crontab)
See crontab(1) for more information
sh-4.1$
I am suspecting this process. any idea about this.
ps axu | grep ‘bash -s /opt/gitlab’
git 7384 1.0 0.0 11344 1288 ? S 10:28 0:07 bash -s /opt/gitlab/
OK, if it’s not letting you view it, check under /var/spool/cron and see if a file exists here for git. If so then you can view that file using cat, eg:
cat /var/spool/cron/git
assuming that the file is called git. I have differently under Debian, /var/spool/cron/crontabs/ and a file would exist for the git user here for example. So you might find yours similar like my example above. If the /var/spool/cron/git file exists, then you can just delete it.
The bash process doesn’t exist on my system, so this is obviously part of the problem.
Either way, once you’ve managed to stop the processes from starting, you need to upgrade your Gitlab.
As for the cron error, maybe git is blocked in /etc/cron.deny or you need to add it to /etc/cron.allow. But using the commands to get to /var/spool/cron as root will get around it and allow is to delete it without editing those files.
You need to stop those. From the post I linked before, the open bash process is constantly downloading something from the internet which is overriding everything you are trying to achieve.
I never had this infection, but you will have to search your server and find everything related to this and stop it permanently. If not cron, then it is picking it up somewhere else.
Just out of interest, check in /opt/gitlab if there are any .bashrc files or something. I’m curious as to why bash -s /opt/gitlab so I have a feeling maybe it’s reading a .bashrc file in /opt/gitlab.
Eg:
ls -lha /opt/gitlab
and see what it lists. Then we can work out if something is there that shouldn’t be.
