Just like other people reported, one repo started to fail the repo check. Gitlab housekeeping does not work.
E, [2022-06-20T10:41:48.849915 #651] ERROR -- : xyz/xyz: Could not fsck repository: error: Could not read 1ae7c8a4c297174103f0c53e57cd709df6692b40
I tried to get rid of the commit in various ways, but nothing works.
What is the official way to deal with this?
Kind regards
Dorian
Hi,
I assume GitLab Housekeeping refers to the Repository Checks in the administration UI. The docs provide more instructions with commands to be run on the CLI to analyze and fix further.
Which other ways did you try to fix the problem, and what was the output? 
Cheers,
Michael
Hi,
actually “housekeeping” refers to the dialog “Repository” → “General” → “Housekeeping”, where I can run “a number of housekeeping tasks”.
The admin repository check itself surfaced a somehow orphaned/dangling commit. My hope was that ‘housekeeping’ runs a ‘git gc’ and gets rid of that.
I googled the error and tried some https://stackoverflow solutions like
git reflog expire --expire=now --all
git gc --prune=now
A local git gc solves the problem in my local copy but cannot be pushed.
I also tried to inspect the bare repository on the server site but had troubles finding the right repository (due to hashed names).
Hi,
thanks for the explanation about housekeeping. I do not have a self-managed instance at hand and wasn’t sure if I understood it correctly.
Regarding the CLI checks, it can be tricky to find the project name from the hashed name. There is a link in the docs pointing to the admin interface and rails console and a few commands to extract the hash from the given project name. Can you try these steps and see if you can find the hash name from either the Admin projects view, or the rails console?
To look up a project’s hash path in the Admin Area:
- On the top bar, select Menu > Admin.
- On the left sidebar, select Overview > Projects and select the project.
With knowing the hashed name and path, execute this git CLI command on the server, replace $HASHPATH with the actual value.
sudo /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/$HASHPATH.git fsck
Unfortunately fixing the repository integrity on a cloned client repository and pushing to the server won’t work, that’s a limitation of Git itself.
Cheers,
Michael
Hi,
thanks for the efforts and hint(s).
sudo /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (5838/5838), done.
Verifying commits in commit graph: 100% (11/11), done.
Verifying commits in commit graph: 100% (376/376), done.
error: Could not read 1ae7c8a4c297174103f0c53e57cd709df6692b40
failed to parse commit 1ae7c8a4c297174103f0c53e57cd709df6692b40 from object database for commit-graph
error: Could not read 211ca05319818652cf0a1a6d405e7e6b3b3f778d
failed to parse commit 211ca05319818652cf0a1a6d405e7e6b3b3f778d from object database for commit-graph
error: Could not read 2882420a19004c7d1f7424e90e3c0058ab8eac69
failed to parse commit 2882420a19004c7d1f7424e90e3c0058ab8eac69 from object database for commit-graph
error: Could not read 294580bb8c78121b860fbf5f2cb92ddd4a1984e0
failed to parse commit 294580bb8c78121b860fbf5f2cb92ddd4a1984e0 from object database for commit-graph
error: Could not read 4347095d448e7776c30887518d95de5e749a8e4d
failed to parse commit 4347095d448e7776c30887518d95de5e749a8e4d from object database for commit-graph
error: Could not read 45607ca276f32c3ef020e1311636187ec6b9567f
failed to parse commit 45607ca276f32c3ef020e1311636187ec6b9567f from object database for commit-graph
error: Could not read 4fc64ad94118d5b50921fa11f2f31cbc4ecd9975
failed to parse commit 4fc64ad94118d5b50921fa11f2f31cbc4ecd9975 from object database for commit-graph
error: Could not read 5c8c4b0ed58930c46eaf1099e5b8c66de0dd7ea2
failed to parse commit 5c8c4b0ed58930c46eaf1099e5b8c66de0dd7ea2 from object database for commit-graph
error: Could not read 5d0a5217d95d27dd9d3abefb35250ec69f77dc75
failed to parse commit 5d0a5217d95d27dd9d3abefb35250ec69f77dc75 from object database for commit-graph
error: Could not read 743b4ef2210c0828dc2f36e99a594020c912a308
failed to parse commit 743b4ef2210c0828dc2f36e99a594020c912a308 from object database for commit-graph
error: Could not read 88417243d02012cbb4a59722d2b2a19f818bdf63
failed to parse commit 88417243d02012cbb4a59722d2b2a19f818bdf63 from object database for commit-graph
error: Could not read a8753df44512452e926c1adec46fd1fa8cf1f86d
failed to parse commit a8753df44512452e926c1adec46fd1fa8cf1f86d from object database for commit-graph
error: Could not read c9e4ecebd6d2c46e57f85bcce793a929df957961
failed to parse commit c9e4ecebd6d2c46e57f85bcce793a929df957961 from object database for commit-graph
error: Could not read d3768f6256984d4abfc93cc6ef55580f90420571
failed to parse commit d3768f6256984d4abfc93cc6ef55580f90420571 from object database for commit-graph
error: Could not read e5a7263b618d695497c6a4cfc319113f30f23208
failed to parse commit e5a7263b618d695497c6a4cfc319113f30f23208 from object database for commit-graph
Verifying commits in commit graph: 100% (1896/1896), done.
Best wishes,
Dorian
Hi,
thanks, we are one step further with a new error message to search for
I found error: Could not read from object database for commit-graph (#2359) · Issues · GitLab.org / gitaly · GitLab which discusses the error message and potential workarounds, in GitLab projects failed their last repository check - #3 by DrrMrr
Before continuing, I’d suggest taking a backup of the git repository storage. It is a bit like operating on the open heart, although with git commands and no direct file handle manipulation. It would be bad if something breaks the repository and there’s no backup to rolback.
Can you try to run fsck, gc, fsck to see if that helps? Commands for easier copy-paste:
sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git fsck
sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git gc
sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git fsck
error: Could not read from object database for commit-graph (#2359) · Issues · GitLab.org / gitaly · GitLab has more suggestions for the commit graph fixes itself.
sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git fsck --lost-found
sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git commit-graph write
sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git reflog expire --expire=now --all
sudo -u git /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/96/06/96061e92f58e4bdcdee73df36183fe3ac64747c81c26f6c83aada8d2aabb1864.git gc --prune=now
And a last comment with new workarounds to try. error: Could not read from object database for commit-graph (#2359) · Issues · GitLab.org / gitaly · GitLab
I’d appreciate if you can test and report back whether it fixes your problems. I’ll link this forum topic in the issue, feel free to comment there as well
Cheers,
Michae l
Hi again,
thanks for preparing the statements.
I also had in mind to run a manual gc, and with some faith in our backup I just did.
As expected, it removed all orphaned commits and solved the issue.
However, I am puzzled that the housekeeping feature is not cleaning this up. Also finding the actual bare repos seems to be a little bit complicated.
Anyways, thanks a lot for your effort, the quick responses and the provided solution.
Much appreciated!
Cheers,
Dorian
Hi,
thanks for confirming that the commands worked After reading the issue with the linked workarounds, I feel that newer Git options may have introduced this behavior, needing both, a housekeeping command addition, and a documentation update. Though, I don’t know about performance impacts or other side effects. I have added a note into error: Could not read from object database for commit-graph (#2359) · Issues · GitLab.org / gitaly · GitLab
Cheers,
Michael
FYI these commands, as written, e.g. sudo /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/lots_of_chars.git fsck broke our ability to push to the repository because they changed the ownership of some files/folders to root instead of git.
drwxr-sr-x 2 git git 4.0K Sep 12 17:31 info
-rw-r--r-- 1 git git 1.7K Aug 30 08:01 language-stats.cache
drwxr-sr-x 4 root git 4.0K Sep 13 10:58 lost-found
drwxr-sr-x 188 git git 4.0K Sep 13 11:26 objects
-rw-r--r-- 1 git git 77K Sep 13 07:51 packed-refs
drwxr-sr-x 5 git git 4.0K Sep 11 16:36 refs
and a whole bunch of folders inside objects and also info/refs.
Users started getting errors when pushing ! [remote rejected] local_branch -> remote_branch (unable to migrate objects to permanent storage).
I fixed it by running a few chown -R git:git affected_folder commands (as root).
I believe there is a way to run these git gc and fsck commands and have them run as the git user, correct?
@carrchr I believe as per the docs: Repository checks | GitLab it has an example using sudo -u git which would get around that.
I have a funny feeling when I did it, that I did it slightly differently, so:
su git -s /bin/bash
/path/to/git -C /path/to/repository fsck
which would also work without the use of sudo. You don’t need to add the -s /bin/bash, it can be without this, but then it will use sh as the shell instead of bash.
Sorry about that. I did not test the commands before replying, and missed the -u git parameters in the documentation. Thanks for flagging, I have edited my comment above.