Gitlab Omnibus installation not displaying repositories following upgrade from 14.4.1-ee to 14.5.0-ee

Hello,

We’re hosting the gitlab omnibus application on Amazon Linux 2 instances in AWS and our architecture consists of a frontend, praefect and gitaly instance that install Gitlab from source. We are also hosting an external PostgreSQL 12 RDS instance of size db.t2.micro.

During a routine upgrade from version 14.4.1-ee to 14.5.0-ee we’ve found that all repositories are showing as empty and displaying the message “The Repository for this project is empty”.

The following commands were run to upgrade on each tier -

yum install -y gitlab-ee
gitlab-ctl reconfigure

We’ve run a series of troubleshooting check such as instance healthchecks using - gitlab-ctl status as well as checking connectivity between instances and database - all of which pass. The application is also showing as updated to 14.5.0-ee in the admin console.

When looking at the application logs on praefect (/var/log/gitlab/praefect/current) we can see the below error message when navigating to repositories. It appears that there’s an grcp issue when praefect is retrieving the project hash from gitaly.

As mentioned above we’ve established that praefect can access gitaly so I don’t think it’s a networking issue.

Any help on this would be very much appreciated.

Thanks,
Matt

{"correlation_id":"01FN8VJHWE8APYVGT6J9G5HJV3","error":"accessor call: route repository accessor: consistent storages: repository \"default\"/\"@hashed/9f/1f/9f1f9dce319c4700ef28ec8c53bd3cc8e6abe64c68385479ab89215806a5bdd6.git\" not found","grpc.code":"NotFound","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"bidi_stream","grpc.method":"HasLocalBranches","grpc.request.deadline":"2021-11-24T11:34:07.002","grpc.request.fullMethod":"/gitaly.RepositoryService/HasLocalBranches","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2021-11-24T11:33:57.703","grpc.time_ms":3.573,"level":"info","msg":"finished streaming call with code NotFound","peer.address":"10.73.36.27:59864","pid":1955,"relative_path":"@hashed/9f/1f/9f1f9dce319c4700ef28ec8c53bd3cc8e6abe64c68385479ab89215806a5bdd6.git","remote_ip":"10.165.17.105","span.kind":"server","system":"grpc","time":"2021-11-24T11:33:57.706Z","username":"root","virtual_storage":"default"}