It’s failing on the post commit hooks. I’m only pushing a tag. Granted, I do have about 50k worth of tags. It started to do this when I upgraded to version 11. Server has 8cpu and 32gb of ram, so it is way over provisioned. It’s killing me trying to keep builds running. Retrying eventually works. Stack trace is below from the production.log
gitlab: 11.4.5
server: centos 7
- GRPC::ResourceExhausted (8:):
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:169:in `receive_and_check_status’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:328:in `block in each_remote_read_then_finish’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `loop’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grpc-1.11.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:325:in `each_remote_read_then_finish’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:107:in `each’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:107:in `flat_map’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:107:in `map_lfs_pointers’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/blob_service.rb:90:in `get_new_lfs_pointers’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git/lfs_changes.rb:10:in `new_pointers’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/checks/lfs_integrity.rb:14:in `objects_missing?’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/checks/change_access.rb:197:in `lfs_objects_exist_check’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/checks/change_access.rb:41:in `exec’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:277:in `check_single_change_access’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:265:in `block in check_change_access!’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:260:in `each’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:260:in `with_index’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:260:in `check_change_access!’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:250:in `check_push_access!’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/git_access.rb:69:in `check’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/api/internal.rb:63:in `block (2 levels) in <class:Internal>’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/head.rb:13:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/mapper.rb:51:in `serve’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:43:in `block in serve’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:30:in `each’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:30:in `serve’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/route_set.rb:817:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/omniauth-1.8.1/lib/omniauth/strategy.rb:190:in `call!’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/omniauth-1.8.1/lib/omniauth/strategy.rb:168:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:101:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:14:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:17:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:11:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/batch-loader-1.2.1/lib/batch_loader/middleware.rb:11:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/etag.rb:24:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/conditionalget.rb:38:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/head.rb:13:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/params_parser.rb:27:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:38:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:16:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/flash.rb:260:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/session/abstract/id.rb:225:in `context’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/session/abstract/id.rb:220:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/cookies.rb:560:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/query_cache.rb:36:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:88:in `run_callbacks’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:778:in `_run_call_callbacks’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:81:in `run_callbacks’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/callbacks.rb:27:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/remote_ip.rb:78:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:38:in `call_app’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:20:in `block in call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:68:in `block in tagged’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:26:in `tagged’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:68:in `tagged’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:20:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:18:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/request_id.rb:21:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/methodoverride.rb:22:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/runtime.rb:18:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/lock.rb:17:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/sendfile.rb:113:in `call’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/rack.rb:51:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:518:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/application.rb:165:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `public_send’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `method_missing’
- /home/gitlab/gitlab_server/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:10:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:66:in `block in call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:50:in `each’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:50:in `call’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>’
- /opt/gitlab/embedded/bin/unicorn:23:in `load’
- /opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors’
- /home/gitlab/gitlab_server/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>’
- /opt/gitlab/embedded/bin/bundle:23:in `load’
- /opt/gitlab/embedded/bin/bundle:23:in `<main>’