Problems with fetching repos via ssh after updated from 16.1.5 to 16.3.3

Hi Team,

I updated my Institution’s Gitlab-EE from 16.1.5 to 16.3.3 which is an Omnibus installation.

Since then when trying to fetch the repos via ssh is resulting in following error -

exec request failed on channel 0 fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists

Also git fetch sometimes throwing this.

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f3a63cae3d7 m=26 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: g 0: unknown pc 0x7f3a63cae3d7
stack: frame={sp:0x7f3a037fd938, fp:0x0} stack=[0x7f3a02ffe2a0,0x7f3a037fdea0)
0x00007f3a037fd838:  0x0000000000000000  0x0000000000000000 ```

However, the repositories where git remote is set to http, they are working fine. Did I miss something during the update?

Could you please help?

Thanks,

Best Regards,

I do not see much in the logs. In the gitlay logs I see following -

{"command.count":1,"command.cpu_time_ms":3,"command.inblock":0,"command.majflt":0,"command.maxrss":373976,"command.minflt":505,"command.oublock":0,"command.real_time_ms":135,"command.spawn_token_fork_ms":0,"command.spawn_token_wait_ms":0,"command.system_time_ms":3,"command.user_time_ms":0,"component":"gitaly.UnaryServerInterceptor","correlation_id":"4c3d996e3e4357ec43468df391baae95","error":"fetch remote: "Permission denied (publickey,gssapi-keyex,gssapi-with-mic).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n": exit status 128","grpc.code":"Internal","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.meta.method_operation":"mutator","grpc.meta.method_scope":"repository","grpc.meta.method_type":"unary","grpc.method":"FetchRemote","grpc.request.deadline":"2023-09-16T00:12:06.996","grpc.request.fullMethod":"/gitaly.RepositoryService/FetchRemote","grpc.request.glProjectPath":"jenny.tran/r-3-courses","grpc.request.glRepository":"project-3246","grpc.request.payload_bytes":4192,"grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2023-09-15T18:12:06.996","grpc.time_ms":137.661,"level":"error","msg":"finished unary call with code Internal","peer.address":"@","pid":70414,"span.kind":"server","system":"grpc","time":"2023-09-15T16:12:07.134Z"}

Also the Gitlab is installed on the physical machine whose resources are -

CPU - 64 (Only few CPUs are occupied, rest seems fine)

Memory - 376G (only 40G used)

In /var/log/secure I see -

error: do_exec_no_pty: fork: Resource temporarily

And Gitlab-ctl restart help but like for 10 mins and then we are back to the error. What is strange is everything was working fine until the update.

Thanks,

Best Regards,

I recently updated my Institution’s Gitlab-EE from version 16.1.5 to 16.3.3 using an Omnibus installation method. Since then, I’ve been encountering an issue when trying to fetch repositories via SSH, which results in the following error:

exec request failed on channel 0
fatal: Could not read from the remote repository.
Please make sure you have the correct access rights and the repository exists.

I’ve also noticed that git fetch sometimes throws the following error:

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f3a63cae3d7 m=26 sigcode=18446744073709551610

Interestingly, repositories where the git remote is set to HTTP seem to be working fine. I’m wondering if there might be something I missed during the update. Your assistance in resolving this issue would be greatly appreciated.