GitLab projects failed their last repository check

How to Use GitLab
1

We have received one time email alert from our gitlab instance recently i.e Sunday 5:30 AM IST on spe-16th.

This alert included the following information. Can you please suggest what is this alert notification all about and how to troubleshoot this issue.We could not find any error details in repocheck.log file from Gitlab UI.

Alert:
Subject: GitLab Admin | 4 projects failed their last repository check
4 projects failed their last repository check.

1 Like
2

I too have a repository that is reporting a failure quite often. We are running GitLab CE 10.7.3 on-prem and there is one repo that keeps failing it’s repository check.

One project has failed it’s repository check
See the affected projects in the GiLab admin panel
You are receiving this message because you are a GitLab administrator for https://my.gitlab.install

I’ve checked the logs in the admin panel and everything comes back clean. I even cloned the repo in question and ran some Git checking/cleanup tools and there were no problems.

We have quite a few repos in this GitLab install but this is the only one that is reporting any problems. What can I do to find this issue and how can it be fixed?

3

Hi,

Below are the instruction of how we solved an issue in GitLab docker environment

  1. find all repositories that failed on link “See the affected projects in the GitLab admin panel” that you’ve received from GitLab
    http://yourgitlab_link/admin/projects?last_repository_check_failed=1

  2. Check /var/log/gitlab/gitlab-rails/repocheck.log file that is generated when “Trigger repository check” is triggered in project admin area. If file is empty then trigger repository check manually again. This step is not so important even though the file is empty or not, because you will get similar output with command fsck described little lower.

  3. open specific project
    Menu->Admin->Projects->your project
    and search for “Gitaly relative path”
    @hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git

  4. go to docker console (connect to container) and run fsck command on that specific repository
    (see more instructions on Repository checks | GitLab)
    command:
    /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git fsck
    and check if there are any errors like

  • error: Could not read 098b53ffbe581e25b…
  • failed to parse commit 098b53ffbe581e25b… from object database for commit-graph
  • …

  1. now run garbage collector gc with
    /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git gc

  2. and check again repository with fsck
    /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git fsck
    and errors and fails should be cleared

  3. run again “Trigger repository check” in check if repository check has passed successfully. If it was successful then this repository shouldn’t be in repository list that have failed to pass the repository check (See the affected projects in the GitLab admin panel) or http://yourgitlab_link/admin/projects?last_repository_check_failed=1

7 Likes
4

Great answer, that helped in my case.
I would just add a small note: git gc may create new files, so ensure you switched to the user gitlab runs as!
If you forget that, you will get permission denied when commiting files - but luckily chown -R fixes that.

5

Thank you very much! It helped me to sort out the problem :slight_smile:

6

To add onto this, if using docker compose the commands can be run with the correct user as follows:

docker compose exec web su - git -c "/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/ff/5a/ff5a1ae012afa5d4c889c50ad427aaf545d31a4fac04ffc1c4d03d403ba4250a.git fsck"
docker compose exec web su - git -c "/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/ff/5a/ff5a1ae012afa5d4c889c50ad427aaf545d31a4fac04ffc1c4d03d403ba4250a.git gc"
docker compose exec web su - git -c "/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/ff/5a/ff5a1ae012afa5d4c889c50ad427aaf545d31a4fac04ffc1c4d03d403ba4250a.git fsck"