Unable to fork repos in my self-hosted gitlab server

The repository could not be imported.

Every fork attempt has failed: Unable to fork project 58 for repository @hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8 → @hashed/62/08/6208ef0f7750c111548cf90b6ea1d0d0a66f6bff40dbef07cb45ec436263c7d6: Failed to create fork repository. Please try again.

unable to fork any repository

Steps to reproduce

1 - Click on fork button
2 - enter new namespace for forked project
3 - click enter

Configuration

Provide screenshots from the GitLab UI showing relevant configuration, if applicable.
On self-managed instances, add the relevant configuration settings or changes.

Versions

Self managed gitlab server on RHEL8
GitLabv16.9.1-ee

GitLab Shell14.33.0

GitLab Workhorsev16.9.1

GitLab APIv4

GitLab KASv16.9.2

Ruby3.1.4p223

Rails7.0.8

PostgreSQL (main)13.13

PostgreSQL (ci)13.13

Redis7.0.15

Check account user has necessary permission to fork repositories. and also check the Gitlab log file for any error.

hello,

Accounts used are admin.
I found some logs in gitaly :
{“component”:“gitaly.UnaryServerInterceptor”,“correlation_id”:“01HQSXP3776CAZ4C433FYY9KBX”,“grpc.code”:“OK”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regula
r”,“grpc.meta.method_operation”:“accessor”,“grpc.meta.method_scope”:“repository”,“grpc.meta.method_type”:“unary”,“grpc.method”:“RepositoryExists”,“grpc.request.deadline”:“2024-02-29T08:54:05.008”,“grpc.reques
t.fullMethod”:“/gitaly.RepositoryService/RepositoryExists”,“grpc.request.glProjectPath”:“test/ansible_sudoers”,“grpc.request.glRepository”:“project-58”,“grpc.request.payload_bytes”:144,“grpc.req
uest.repoPath”:“@hashed/62/08/6208ef0f7750c111548cf90b6ea1d0d0a66f6bff40dbef07cb45ec436263c7d6.git”,“grpc.request.repoStorage”:“default”,“grpc.response.payload_bytes”:0,“grpc.service”:“gitaly.RepositoryServic
e”,“grpc.start_time”:“2024-02-29T08:53:55.738”,“grpc.time_ms”:0.113,“level”:“info”,“msg”:“finished unary call with code OK”,“pid”:1705,“remote_ip”:“179.114.29.33”,“span.kind”:“server”,“system”:“grpc”,“time”:"
2024-02-29T07:53:55.739Z",“user_id”:“2”,“username”:“test”}
{“component”:“gitaly.UnaryServerInterceptor”,“correlation_id”:“01HQSXP3776CAZ4C433FYY9KBX”,“error”:“repository not found”,“error_metadata”:{“relative_path”:“@hashed/62/08/6208ef0f7750c111548cf90b6ea1d0d0a66f6
bff40dbef07cb45ec436263c7d6.git”,“storage_name”:“default”},“grpc.code”:“NotFound”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_opera
tion”:“accessor”,“grpc.meta.method_scope”:“repository”,“grpc.meta.method_type”:“unary”,“grpc.method”:“FindDefaultBranchName”,“grpc.request.deadline”:“2024-02-29T08:54:05.044”,“grpc.request.fullMethod”:“/gital
y.RefService/FindDefaultBranchName”,“grpc.request.glProjectPath”:“test/ansible_sudoers”,“grpc.request.glRepository”:“project-58”,“grpc.request.payload_bytes”:144,“grpc.request.repoPath”:“@hashed
/62/08/6208ef0f7750c111548cf90b6ea1d0d0a66f6bff40dbef07cb45ec436263c7d6.git”,“grpc.request.repoStorage”:“default”,“grpc.response.payload_bytes”:0,“grpc.service”:“gitaly.RefService”,“grpc.start_time”:“2024-02-
29T08:53:55.774”,“grpc.time_ms”:0.123,“level”:“info”,“msg”:“finished unary call with code NotFound”,“pid”:1705,“remote_ip”:“179.114.29.33”,“span.kind”:“server”,“system”:“grpc”,“time”:“2024-02-29T07:53:55.774Z
“,“user_id”:“2”,“username”:“test”}
{“component”:“gitaly.UnaryServerInterceptor”,“correlation_id”:“01HQSZ1AQTPD6YY4Y59M9N2RXZ”,“diskcache”:“52e97b0d-ecfc-45aa-831e-6e0553336fb8”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-sidekiq”,“grpc.meta.method_operation”:“mutator”,“grpc.meta.method_scope”:“repository”,“grpc.meta.method_type”:“unary”,“grpc.method”:“CreateFork”,“grpc.request.deadline”:“2024-02-29T15:19:09.683”,“grpc.request.fullMethod”:”/gitaly.RepositoryService/CreateFork”,“grpc.request.glProjectPath”:“test/bastion-openshift”,“grpc.request.glRepository”:“project-59”,“grpc.request.repoPath”:“@hashed/3e/1e/3e1e967e9b793e908f8eae83c74dba9bcccce6a5535b4b462bd9994537bfe15c.git”,“grpc.request.repoStorage”:“default”,“grpc.service”:“gitaly.RepositoryService”,“grpc.start_time”:“2024-02-29T09:19:09.683”,“level”:“info”,“msg”:“diskcache state change”,“pid”:1705,“remote_ip”:“179.114.29.33”,“span.kind”:“server”,“system”:“grpc”,“time”:“2024-02-29T08:19:09.692Z”,“user_id”:“2”,“username”:“test”}
{“command.count”:1,“command.cpu_time_ms”:3,“command.inblock”:0,“command.majflt”:0,“command.maxrss”:348456,“command.minflt”:176,“command.oublock”:48,“command.real_time_ms”:5,“command.spawn_token_fork_ms”:0,“command.spawn_token_wait_ms”:0,“command.system_time_ms”:2,“command.user_time_ms”:1,“component”:“gitaly.UnaryServerInterceptor”,“correlation_id”:“01HQSZ1AQTPD6YY4Y59M9N2RXZ”,“error”:“creating fork: fetching source repo: exit status 128, stderr: "/var/opt/gitlab/gitaly/run/gitaly-1705/gitaly-ssh upload-pack: /var/opt/gitlab/gitaly/run/gitaly-1705/gitaly-ssh: Permission denied\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"”,“grpc.code”:“Internal”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-sidekiq”,“grpc.meta.method_operation”:“mutator”,“grpc.meta.method_scope”:“repository”,“grpc.meta.method_type”:“unary”,“grpc.method”:“CreateFork”,“grpc.request.deadline”:“2024-02-29T15:19:09.683”,“grpc.request.fullMethod”:“/gitaly.RepositoryService/CreateFork”,“grpc.request.glProjectPath”:“jfourniols/ansible-bastion-openshift”,“grpc.request.glRepository”:“project-59”,“grpc.request.payload_bytes”:291,“grpc.request.repoPath”:“@hashed/3e/1e/3e1e967e9b793e908f8eae83c74dba9bcccce6a5535b4b462bd9994537bfe15c.git”,“grpc.request.repoStorage”:“default”,“grpc.response.payload_bytes”:0,“grpc.service”:“gitaly.RepositoryService”,“grpc.start_time”:“2024-02-29T09:19:09.683”,“grpc.time_ms”:9.079,“level”:“error”,“msg”:“finished unary call with code Internal”,“pid”:1705,“remote_ip”:“179.114.29.33”,“span.kind”:“server”,“system”:“grpc”,“time”:“2024-02-29T08:19:09.692Z”,“user_id”:“2”,“username”:“test”}

Forking in progress

git clone --bare http://*****:*****@gitlab.test.local/satellite/ansible_sudoers.git

Please wait while we import the repository for you. Refresh at will.

I have my /var filesystem with noexec option…