Unable to click Merge button for Merge Requests, and crond is down

I just recently updated my Private GitLab CE to 13.12.4, the latest version.

Unable to click Merge button in Merge Request

When I open a Merge Request, and go down to where the Merge button is, it is greyed out and all I see is the message Checking if merge request can be merged… continuously without end.

I can approve a Merge Request, no issues there.

I also tried entering the GitLab Website through multiple browsers:

• Firefox
• Google Chrome
• Microsoft Edge

Merge button is greyed out and all I see is the message Checking if merge request can be merged… continuously without end.

crond is down

I also noticed that, in Terminal, when I run gitlab-ctl status, everything is up … except crond:

down: crond: 409s, normally up; run: log: (pid 1450) 1234s

When I try restarting crond, the process is successful

# gitlab-ctl restart crond
ok: run: crond: (pid 28954) 0s

… but checking the status again shows it as down:

# gitlab-ctl status
down: crond: 1s, normally up, want up; run: log: (pid 1450) 2102s

I’ve ran a restart (gitlab-ctl restart) and a reconfigure (gitlab-ctl reconfigure). Neither reports any errors, but crond is still down.

Upon the restart, crond will come up … but will immediately go down again.

I’ve also tried restarting my GitLab VM, but crond is still down.

When I run head /var/log/gitlab/crond/current, I keep seeing the following message every few seconds over and over:

==> /var/log/gitlab/crond/current <==
2021-06-17_19:39:44.27476 time="2021-06-17T19:39:44Z" level=error msg="unknown flag `no-auto'"

Running sudo gitlab-ctl tail | grep error, I receive more errors and information:

==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
2021-06-17_19:53:18.70815 time="2021-06-17T19:53:18Z" level=error msg="unknown flag `no-auto'"
...
2021-06-17_19:55:58.77641 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-06-17T19:55:58.776380Z","queryType":"random_walk_with_error"}
2021-06-17_19:55:58.77649 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-06-17T19:55:58.776450Z","queryType":"server_error_500"}
2021-06-17_19:55:58.81158 t=2021-06-17T19:55:58+0000 lvl=eror msg="Failed to read plugin provisioning files from directory" logger=provisioning.plugins path=/var/opt/gitlab/grafana/provisioning/plugins error="open /var/opt/gitlab/grafana/provisioning/plugins: no such file or directory"
2021-06-17_19:54:11.85015 LOG:  configuration file "/var/opt/gitlab/postgresql/data/postgresql.conf" contains errors; unaffected changes were applied
2021-06-17_19:56:03.39790 level=error ts=2021-06-17T19:56:03.397Z caller=manager.go:314 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2021-06-17_19:56:08.52511 level=error ts=2021-06-17T19:56:08.524Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-nodes
2021-06-17_19:56:08.52538 level=error ts=2021-06-17T19:56:08.525Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-cadvisor
2021-06-17_19:56:08.52555 level=error ts=2021-06-17T19:56:08.525Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-pods

Right now I’m thinking the two problems are related. Any recommendations on how to fix these?

So here’s something that semi-fixed the problem:

1. Closed the Merge Request
2. Re-opened the Merge Request

Now, no more Checking if merge request can be merged… continuously without end! Refreshing the page doesn’t even show the continuous message anymore!

However, I am getting told There are merge conflicts with the only option to Merge locally.

I would still like to figure out WHY this Issue is occurring, and why crond is down.

Another update. and I got Good News and Bad News.

Good News: The original Merge Requester decided to do the following:

1. Re-pull master (he was about 17 commits behind)
2. Re-add the files to the Project he plans to branch
3. Create a New Branch with his changes
4. Pushed changes to the New Branch
5. Created a New Merge Request

Now, at first the New Merge Request had the Checking if merge request can be merged… up for awhile … but after a few mins, that stopped and we have the green Merge button (no merge conflicts!). There really isn’t any new changes, but clearly the Merge Requester being behind master was a real problem.

The Merge Request hasn’t been merged yet though, so we don’t know if there will be any problems.

Bad News: The same Merge Requester is setting up a new Merge Request from another Branch … and now every time he tries to create a merge request, he gets a 500 Error page in GitLab.

Now Server-wise (we host our GitLab on a Private Server VM): no disk space issues, CPU Usage is low, Memory Usage is only at 50%, so not any issues there. crond (and all the previous errors I shared) are still all there, but clearly this 500 Error page is brand new.

Could it be that merging the previous Merge Request, and then him re-creating a Merge Request with his new changes, may fix things? No idea.

Once the previous Merge Request is merged (and assuming no issues there), I’m going to suggest the Merge Requester to try re-creating his content after another master pull, new branch, push content and then try creating a Merge Request. If that doesn’t resolve the issue, I’m going to backup and restart GitLab. Not sure what else to do.

Merge locally comes up time-to-time when something has changed on both the source and target branch. This means that git cannot determine which change should be kept. Visual Studio Code has a great merge conflict feature that can help with a local merge.

Wish I could help with the error, but I haven’t seen it before. Maybe try toggling the merge types in Settings → Merge Method.

Thank you for responding @byarbrough

Yeah, that issue definitely seems related to the fact the original Merge Requester was 17 commits behind master. Once he pulled master and then implemented his changes, created a new branch and a new Merge Request, all was fine as I mentioned here … well, somewhat. I’ll explain later.

Nice to know, but irrelevant: not all our files are C++. In fact, vast majority of our files aren’t. I’m part of an Unreal Engine game development team. The vast majority of our files are in a proprietary format called .uasset, which are basically 2D Images and 3D Models interpreted by the Engine.

Let me share our current Merge Request settings here. Any recommendations on what you think we should try changing?

MergeRequest_Settings1228×1520 65.8 KB

Unfortunately, I got Bad News: even after another restart …

1. crond is still down
2. New Merge Requests still receive the Checking if merge request can be merged… loop.

The only solution to the 2nd one seems to be Closing and Re-opening the Merge Request. This occurs even if there are no Merge Conflicts, so it definitely doesn’t only occur with Merge Requests with Merge Conflicts.

Please not I have upgraded to 13.12.5 this morning. crond is still down and I have yet to test a Merge Request with the recent update.

EDIT: I’ve also shared my issues with Merge Requests and crond here. Apparently the two are completely unrelated. Since that GitLab Issue I linked is related to crond, then I’d rather focus on addressing my Merge Request issues here.

Performed a tail /var/log/gitlab/sidekiq/current and the results can be found here.

Also performed a less /var/log/gitlab/sidekiq/current | grep error and the results can be found here (I’ve modified the Team and Project names for privacy purposes). The main error messages I see constantly coming up in sideqik are the following:

• Error creating pipeline
• Missing CI config file
• Reference not found
• Deadline Exceeded
• Heartbeat thread error: Error connecting to Redis on /var/opt/gitlab/redis/redis.socket (Errno::ENOENT)

I hope someone here has some ideas on how to resolve this.

So @stanhu resolved my problem in this GitLab Issue!

Effectively the solution was to remove --no-auto \ from /opt/gitlab/sv/crond/run and then performed gitlab-ctl restart crond. crond has been staying in the run state since then!

root@gitlab:/opt/gitlab/sv/crond# gitlab-ctl status
run: crond: (pid 20366) 232s; run: log: (pid 1399) 99784s

@stanhu doesn’t believe this is related to the Merge button issue, and until my Team makes another Merge Request, I can’t say the problem has been resolved.

I believe strongly that the changes I’ve made here have assisted the Merge button issues I’ve had with GitLab. I’ve have yet to have the Checking if merge request can be merged… message reappearing. I don’t want to call this issue as resolved yet, still want to keep observing for it right now.

Seriously, no issues with anyone Pushing, Creating Merge Requests, or Pulling to/from GitLab. You can also click on any of our large Commits and not receive a 500 error as was happening before. Looks to me the majority of my GitLab issues were related to Puma and Gitaly not having enough resources put behind them, and by default GitLab limits you unless you edit gitlab.rb.

Moving forward, I do think there should be easier ways to put more processing power behind different tools in GitLab, and perhaps better suggestions in the Logs.

This issue of mine has not been resolved yet. See this post for details.

Unless the solution could be, again, to increase the Puma Worker Timeout and Gitaly Timeouts even higher (past 5 minutes).

EDIT: Actually, I just refreshed the webpage and now a Green Merge button has appeared nearly instantaneously after the Merge Request page reloaded. So I’m going to say this issue has been resolved, but I really would like feedback on how this process could be improved.