Could not read from remote repo

Hi guys,

I am getting an error where we cant read the repos anymore. Any advice would be appreciated.

GitLab Community Edition 13.9.2


Can you try clearing the redis cache as per this link: Maintenance Rake tasks | GitLab

Hi iwalker,

Thanks again for your replies, really appreciate the support!

Because of the issues Gitlab 502 all sockets connection refused - #11 by michael.davenport all Rake commands I run respond with “Killed”, so this isn’t something I can do until the other issues are resolved.


Ok I tried that, and it did not return an error this time.


Hi Mark,

I have done that but this is what I am seeing on all repos:

And here is the latest run on the rake check command:

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.16.1 ? ... OK (13.16.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 25
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Again many thanks Iwalker.

==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
{“correlation_id”:“EwtZYWbDVy4”,“duration_ms”:286,“error”:“A repository for this project does not exist yet.”,“level”:“error”,“method”:“POST”,“msg”:“Internal API error”,“status”:404,“time”:“2021-11-30T14:14:02Z”,“url”:“http://unix/api/v4/internal/allowed”}

Hi @michael.davenport . :wave:

These error messages would suggest your repository data has been corrupted on disk in a way that it’s no longer recognized as a Git repository.

Is this affecting all repositories?
If so, I recommend restoring from a backup captured before you started seeing this error.

Hi Greg,

Thanks for your reply.

Yes it appears to be all repos, I am not even sure if we have a backup and/or if we have ever restored from backup.

Something I have tried is to push the button to create an empty repo on one of the existing repos. Then I have grabbed the code from my local and tried to push the changes to it, however it doesn’t recognize any of the changes.

It’s like the database knows the repo should have these files, but it cant see them on the server, but trying to push those changes in git and they are not visible as diffs.


Ok there are some backups from Feb 2021, if I restore them it sounds like it will overide the database, but do you know if this will recover the repos as well please Greg?

Is there anyway I can recover just the files from this? and leave the database intact please?