Problem to solve
Hi all, I manage an instance of Gitlab for an academic institution.
Recently, at seemingly random, users will begin getting an “Internal API Error 500” on any command-line action they take.
On my initial dive into logs, I found that its caused by the gitaly socket occasionally returning Permission Denied, but I cannot for the life of me understand why.
The only reliable way to fix this has been to restart the Gitlab instance, but it’s been happening enough where it’s been significantly hampering peoples experience with the application and causing delays for everyone involved.
Please advise.
Steps to reproduce
No clue, currently.
LOGS
Here is one such failures I was able to pull from logs.
{
"severity": "ERROR",
"time": "2025-04-28T19:44:51.082Z",
"correlation_id": "01JSYYWJBFCWDPWCT4X5MYTS54",
"meta.caller_id": "Projects::BlobController#update",
"meta.feature_category": "source_code_management",
"meta.organization_id": 1,
"meta.remote_ip": "10.0.2.100",
"meta.user": "caalicea",
"meta.user_id": 2771,
"meta.client_id": "user/2771",
"exception.class": "Gitlab::Git::CommandError",
"exception.message": "13:update reference: running pre-receive hooks: GitLab: Internal API error (500).",
"exception.backtrace": [
"lib/gitlab/git/wraps_gitaly_errors.rb:52:in `handle_default_error'",
"lib/gitlab/git/wraps_gitaly_errors.rb:23:in `handle_error'",
"lib/gitlab/git/wraps_gitaly_errors.rb:14:in `rescue in wrapped_gitaly_errors'",
"lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'",
"lib/gitlab/git/repository.rb:1031:in `commit_files'",
"app/models/repository.rb:897:in `block in commit_files'",
"app/models/repository.rb:875:in `with_cache_hooks'",
"app/models/repository.rb:897:in `commit_files'",
"app/models/repository.rb:842:in `update_file'",
"app/services/files/update_service.rb:14:in `create_transformed_commit'",
"app/services/files/update_service.rb:10:in `create_commit!'",
"app/services/commits/create_service.rb:30:in `execute'",
"app/controllers/concerns/creates_commit.rb:33:in `create_commit'",
"app/controllers/projects/blob_controller.rb:94:in `update'",
"actionpack (7.0.8.6) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'",
"actionpack (7.0.8.6) lib/abstract_controller/base.rb:215:in `process_action'",
"actionpack (7.0.8.6) lib/action_controller/metal/rendering.rb:165:in `process_action'",
"actionpack (7.0.8.6) lib/abstract_controller/callbacks.rb:234:in `block in process_action'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:118:in `block in run_callbacks'",
"app/controllers/application_controller.rb:505:in `set_current_admin'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:496:in `set_session_storage'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
"lib/gitlab/i18n.rb:114:in `with_locale'",
"lib/gitlab/i18n.rb:120:in `with_user_locale'",
"app/controllers/application_controller.rb:487:in `set_locale'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
"app/controllers/application_controller.rb:480:in `set_current_context'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
"marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
"sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:32:in `block in sentry_around_action'",
"sentry-ruby (5.21.0) lib/sentry/hub.rb:108:in `with_child_span'",
"sentry-ruby (5.21.0) lib/sentry-ruby.rb:499:in `with_child_span'",
"sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:138:in `run_callbacks'",
"actionpack (7.0.8.6) lib/abstract_controller/callbacks.rb:233:in `process_action'",
"actionpack (7.0.8.6) lib/action_controller/metal/rescue.rb:23:in `process_action'",
"actionpack (7.0.8.6) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'",
"activesupport (7.0.8.6) lib/active_support/notifications.rb:206:in `block in instrument'",
"activesupport (7.0.8.6) lib/active_support/notifications/instrumenter.rb:24:in `instrument'",
"activesupport (7.0.8.6) lib/active_support/notifications.rb:206:in `instrument'",
"actionpack (7.0.8.6) lib/action_controller/metal/instrumentation.rb:66:in `process_action'",
"actionpack (7.0.8.6) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'",
"activerecord (7.0.8.6) lib/active_record/railties/controller_runtime.rb:27:in `process_action'",
"actionpack (7.0.8.6) lib/abstract_controller/base.rb:151:in `process'",
"actionview (7.0.8.6) lib/action_view/rendering.rb:39:in `process'",
"actionpack (7.0.8.6) lib/action_controller/metal.rb:188:in `dispatch'",
"actionpack (7.0.8.6) lib/action_controller/metal.rb:249:in `block in dispatch'",
"lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'",
"actionpack (7.0.8.6) lib/action_controller/metal.rb:249:in `dispatch'",
"actionpack (7.0.8.6) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'",
"actionpack (7.0.8.6) lib/action_dispatch/routing/route_set.rb:32:in `serve'",
"actionpack (7.0.8.6) lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'",
"actionpack (7.0.8.6) lib/action_dispatch/routing/mapper.rb:48:in `serve'",
"actionpack (7.0.8.6) lib/action_dispatch/journey/router.rb:50:in `block in serve'",
"actionpack (7.0.8.6) lib/action_dispatch/journey/router.rb:32:in `each'",
"actionpack (7.0.8.6) lib/action_dispatch/journey/router.rb:32:in `serve'",
"actionpack (7.0.8.6) lib/action_dispatch/routing/route_set.rb:852:in `call'",
"gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'",
"omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!'",
"omniauth-saml (2.2.1) lib/omniauth/strategies/saml.rb:86:in `other_phase'",
"omniauth (2.1.0) lib/omniauth/strategy.rb:195:in `call!'",
"omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'",
"flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'",
"flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
"lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/go.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:83:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/ci/job_token/middleware.rb:11:in `call'",
"batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'",
"apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'",
"warden (1.2.9) lib/warden/manager.rb:36:in `block in call'",
"warden (1.2.9) lib/warden/manager.rb:34:in `catch'",
"warden (1.2.9) lib/warden/manager.rb:34:in `call'",
"rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'",
"rack (2.2.10) lib/rack/tempfile_reaper.rb:15:in `call'",
"rack (2.2.10) lib/rack/etag.rb:27:in `call'",
"rack (2.2.10) lib/rack/conditional_get.rb:40:in `call'",
"rack (2.2.10) lib/rack/head.rb:12:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/http/permissions_policy.rb:38:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/http/content_security_policy.rb:36:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
"rack (2.2.10) lib/rack/session/abstract/id.rb:266:in `context'",
"rack (2.2.10) lib/rack/session/abstract/id.rb:260:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/cookies.rb:704:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'",
"activesupport (7.0.8.6) lib/active_support/callbacks.rb:99:in `run_callbacks'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/callbacks.rb:26:in `call'",
"sentry-rails (5.21.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'",
"lib/gitlab/middleware/path_traversal_check.rb:35:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'",
"sentry-ruby (5.21.0) lib/sentry/hub.rb:265:in `with_session_tracking'",
"sentry-ruby (5.21.0) lib/sentry-ruby.rb:412:in `with_session_tracking'",
"sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'",
"sentry-ruby (5.21.0) lib/sentry/hub.rb:59:in `with_scope'",
"sentry-ruby (5.21.0) lib/sentry-ruby.rb:392:in `with_scope'",
"sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'",
"railties (7.0.8.6) lib/rails/rack/logger.rb:25:in `block in call'",
"activesupport (7.0.8.6) lib/active_support/tagged_logging.rb:99:in `block in tagged'",
"activesupport (7.0.8.6) lib/active_support/tagged_logging.rb:37:in `tagged'",
"activesupport (7.0.8.6) lib/active_support/tagged_logging.rb:99:in `tagged'",
"railties (7.0.8.6) lib/rails/rack/logger.rb:25:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:15:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'",
"rack (2.2.10) lib/rack/method_override.rb:24:in `call'",
"rack (2.2.10) lib/rack/runtime.rb:22:in `call'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'",
"rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:44:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/executor.rb:14:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"rack (2.2.10) lib/rack/sendfile.rb:110:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
"gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:22:in `block in call'",
"gitlab-labkit (0.37.0) lib/labkit/context.rb:35:in `with_context'",
"gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:21:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/request_id.rb:26:in `call'",
"actionpack (7.0.8.6) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'",
"railties (7.0.8.6) lib/rails/engine.rb:530:in `call'",
"railties (7.0.8.6) lib/rails/railtie.rb:226:in `public_send'",
"railties (7.0.8.6) lib/rails/railtie.rb:226:in `method_missing'",
"lib/gitlab/middleware/release_env.rb:12:in `call'",
"rack (2.2.10) lib/rack/urlmap.rb:74:in `block in call'",
"rack (2.2.10) lib/rack/urlmap.rb:58:in `each'",
"rack (2.2.10) lib/rack/urlmap.rb:58:in `call'",
"puma (6.5.0) lib/puma/configuration.rb:279:in `call'",
"puma (6.5.0) lib/puma/request.rb:99:in `block in handle_request'",
"puma (6.5.0) lib/puma/thread_pool.rb:389:in `with_force_shutdown'",
"puma (6.5.0) lib/puma/request.rb:98:in `handle_request'",
"puma (6.5.0) lib/puma/server.rb:468:in `process_client'",
"puma (6.5.0) lib/puma/server.rb:249:in `block in run'",
"puma (6.5.0) lib/puma/thread_pool.rb:166:in `block in spawn_thread'"
],
"exception.cause_class": "GRPC::Internal",
"exception.gitaly": "{:storage=>\"default\", :address=>\"unix:/var/opt/gitlab/gitaly/gitaly.socket\", :service=>:operation_service, :rpc=>:user_commit_files}",
"user.username": "caalicea",
"tags.program": "web",
"tags.locale": "en",
"tags.feature_category": "source_code_management",
"tags.correlation_id": "01JSYYWJBFCWDPWCT4X5MYTS54"
}
Digging into grpc.log I find this
W, [2025-04-28T19:44:12.261880 #1369346] WARN -- GRPC: bidi: read-loop failed
W, [2025-04-28T19:44:12.264003 #1369346] WARN -- GRPC: 7:GitLab: Internal API error (500). debug_error_string:{UNKNOWN:Error received from peer unix:/var/opt/gitlab/gitaly/gitaly.socket {created_time:"2025-04-28T19:44:12.260686349+00:00", grpc_status:7, grpc_message:"GitLab: Internal API error (500)"}} (GRPC::PermissionDenied)
W, [2025-04-28T19:44:51.313560 #1369346] WARN -- GRPC: bidi: read-loop failed
W, [2025-04-28T19:44:51.315640 #1369346] WARN -- GRPC: 13:target update: running post-receive hooks: GitLab: http post to gitlab api /post_receive endpoint: Internal API error (500). debug_error_string:{UNKNOWN:Error received from peer unix:/var/opt/gitlab/gitaly/gitaly.socket {grpc_message:"target update: running post-receive hooks: GitLab: http post to gitlab api /post_receive endpoint: Internal API error (500)", grpc_status:13, created_time:"2025-04-28T19:44:51.31214715+00:00"}} (GRPC::Internal)
Versions
Please add an x whether options apply, and add the version information.
- Self-managed (Docker)
-
GitLab.comSaaS - Dedicated
Versions
System information
System:
Current User: git
Using RVM: no
Ruby Version: 3.2.5
Gem Version: 3.5.23
Bundler Version:2.5.11
Rake Version: 13.0.6
Redis Version: 7.0.15
Sidekiq Version:7.2.4
Go Version: unknown
GitLab information
Version: 17.7.7
Revision: 07af4386169
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 14.17
...
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: saml
GitLab Shell
Version: 14.39.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 17.7.7
- default Git Version: 2.47.0
if NVMe yeah shouldn’t be an issue at all with that.