Git push triggers exception in gitlab-rails ; GPRC::DeadlineExceeded (4:Deadline Exceeded)

Hi,

I am running GitLab CE (omnibus package: gitlab-ce-11.2.3-ce.0.el7.x86_64) on CentOS 7.5.

When I push a small commit, everything goes fine, but if I push many commits at once (for example a full project) or a commit that changes hundreds of files, I receive systematically the following error:

remote: GitLab: API is not accessible
To mygitlab.localdomain:mygroup/myrepo.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@mygitlab.localdomain:mygroup/myrepo.git'

The error is reproducible for all GIT repositories hosted on the GitLab server and all git clients version I have:

• git 2.11.0 on Debian 9
• git 2.1.4 on Debian 8
• git 1.7.10.4 on Debian 7

This error is related with the following ruby exception in /var/log/gitlab/gitlab-rails/production.log

GRPC::DeadlineExceeded (4:Deadline Exceeded):
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:169:in `receive_and_check_status'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:328:in `block in each_remote_read_then_finish'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `loop'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `each_remote_read_then_finish'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:107:in `each'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:107:in `flat_map'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:107:in `map_lfs_pointers'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:90:in `get_new_lfs_pointers'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/lfs_changes.rb:10:in `new_pointers'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/checks/lfs_integrity.rb:13:in `objects_missing?'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/checks/change_access.rb:197:in `lfs_objects_exist_check'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/checks/change_access.rb:41:in `exec'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:269:in `check_single_change_access'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:257:in `block in check_change_access!'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:252:in `each'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:252:in `with_index'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:252:in `check_change_access!'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:242:in `check_push_access!'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:65:in `check'
  /opt/gitlab/embedded/service/gitlab-rails/lib/api/internal.rb:53:in `block (2 levels) in <class:Internal>'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:57:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:56:in `block in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:262:in `block in run'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:243:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:313:in `block in build_stack'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/base.rb:31:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/base.rb:31:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/error.rb:37:in `block in call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/error.rb:36:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/error.rb:36:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:227:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/endpoint.rb:221:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router/route.rb:72:in `exec'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router.rb:121:in `process_route'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router.rb:74:in `block in identity'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router.rb:93:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router.rb:72:in `identity'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router.rb:57:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router.rb:137:in `with_optimization'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/router.rb:56:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/api.rb:119:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/api.rb:45:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-1.0.3/lib/grape/api.rb:40:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/mapper.rb:51:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:43:in `block in serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:30:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:30:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/route_set.rb:817:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/omniauth-1.8.1/lib/omniauth/strategy.rb:190:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/omniauth-1.8.1/lib/omniauth/strategy.rb:168:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:97:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:14:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:17:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:11:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/batch-loader-1.2.1/lib/batch_loader/middleware.rb:11:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/etag.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/conditionalget.rb:38:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/head.rb:13:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:38:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:16:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/flash.rb:260:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/session/abstract/id.rb:225:in `context'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/session/abstract/id.rb:220:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/cookies.rb:560:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/query_cache.rb:36:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:38:in `call_app'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:20:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:68:in `block in tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:26:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:68:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:20:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:18:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/request_id.rb:21:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/methodoverride.rb:22:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/runtime.rb:18:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/lock.rb:17:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/sendfile.rb:113:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/rack.rb:51:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:518:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/application.rb:165:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `public_send'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `method_missing'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:10:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:66:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:50:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:50:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
  /opt/gitlab/embedded/bin/unicorn:23:in `load'
  /opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
  /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
  /opt/gitlab/embedded/bin/bundle:23:in `load'
  /opt/gitlab/embedded/bin/bundle:23:in `<main>'

The system never lacks of CPU, memory or disk resources.

It would be very nice, if you can give me a hint to solve this issue.

Best regards

Sebastien

I have finally found a workaround:

My git repositories was stored in a separated path into a NFS share, mounted with the tcp,vers=3,noac,noatime options. It was safer but it creates a performance penalty.

I have remount the NFS share without noac, that means with tcp,vers=3,noac,noatime, and now, gitlab runs three time faster, and then the timeout exception has disappeared.