Missing Branches and 503 Merge Requests

Something strange is going on on my private GitLab Server.

I have 2 Merge Requests, both created 2 days ago. When I click them in GitLab, I receive “500 - Whoops, something went wrong on our end.” All previous Merge Requests (before 2 days ago) I can access.

More importantly, when I go to the Project’s page and click the drop-down to access other branches besides master (our master branch), I receive “An error occurred while getting projects” and have an infinitie spinning “loading” error. I also have developers on my team saying they can’t access any of the branches in our project outside master.

Earlier today, I ran gitlab-ctl tail and saw 1 constant error:
2020-08-05_14:07:57.36737 level=error ts=2020-08-05T14:07:57.366865365Z caller=main.go:717 err="opening storage failed: block dir: \"/var/opt/gitlab/prometheus/data/01E82EJPX9QP38117Z883ZNWS7\": unexpected end of JSON input

By following this guide, I was able to resolve this Prometheus issue (no longer comes up when gitlab-ctl tail is run). However, the 2 503 Merge Requests and the Missing Branches are still in effect.

Desperate to find anything in GitLab’s Logs, I ran sudo gitlab-ctl tail | grep error. Here is a snippet that stood out:

lib/gitlab/git/wraps_gitaly_errors.rb:13:inrescue in wrapped_gitaly_errors’`


{"correlation_id":"akZJLf9i3s4","grpc.meta.auth_version":"v2","grpc.method":"PostUploadPack","grpc.request.fullMethod":"/gitaly.SmartHTTPService/PostUploadPack","grpc.request.glProjectPath":"company/Project","grpc.request.glRepository":"project-37","grpc.request.repoPath":"company/Project.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"company","grpc.service":"gitaly.SmartHTTPService","grpc.start_time":"2020-08-05T15:00:16Z","level":"error","msg":"error: object file ./objects/e9/b1b502818411be3a2d827ce72aee986e2b3763 is empty\\nfatal: loose object e9b1b502818411be3a2d827ce72aee986e2b3763 (stored in ./objects/e9/b1b502818411be3a2d827ce72aee986e2b3763) is corrupt\\n","peer.address":"@","span.kind":"server","system":"grpc","time":"2020-08-05T15:00:16Z"}

(I have obviously replaced the team and project name for security purposes)

So could the problem be with Gitaly? Gitaly isn’t even enabled in /etc/gitlab/gitlab.rb. I literally have # gitaly['enable'] = true (commented out).

Any suggestions on how to resolve my GitLab Issues?

So turns out, the issue was a bad branch: a developer on our team who pushed last deleted his branch and all was well. The 2 Merge Requests now show up properly this time, and the drop-down branches menu now works again. The same developer also re-pushed that previously-corrupted branch and all is fine still.

I believe our GitLab Server lost connection during his branch push. It certainly wasn’t a file size issue, as our GitLab has plenty of free space, but I suspect it was related to our Nginx Reverse Proxy server crashing during the Git Push.

I discovered this branch issue being the cause after going into /var/opt/gitlab/git-data/repositories/team/Project.git/objects/ and manually moving all the 0-byte files to a temporary place, and later getting refs/heads/KP_MenuImprovementsCppFix does not point to a valid object! which was the problem branch in question.