Hello,
after following the gitlab documentation in order to have the gitlab in HA i’m encountering this error: from the GUI when i try for example to commit something:
{"severity":"ERROR","time":"2022-01-06T20:24:54.930Z","correlation_id":"01FRRGZYQ6ABS292PZN9CQXR7V","exception.class":"Gitlab::Git::CommandTimedOut","exception.message":"4:Deadline Exceeded.","exception.backtrace":["lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'","lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:916:in `multi_action'","app/models/repository.rb:827:in `block in multi_action'","app/models/repository.rb:810:in `with_cache_hooks'","app/models/repository.rb:827:in `multi_action'","app/services/files/multi_service.rb:41:in `commit_actions!'","app/services/files/multi_service.rb:13:in `create_commit!'","app/services/commits/create_service.rb:30:in `execute'","lib/api/commits.rb:133:in `block (2 levels) in \u003cclass:Commits\u003e'","lib/api/api_guard.rb:213:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:26:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"user.username":"gianmarco.carrieri","tags.program":"web","tags.locale":"en","tags.feature_category":"source_code_management","tags.correlation_id":"01FRRGZYQ6ABS292PZN9CQXR7V"}```
My setup is with the praefect and gitaly on separated ec2 cluster.
From SSH all works fine (push, create branch etc etc), but when i try to commit a file or do anything from the GUI on repo i encounter in the error.
The Gitlab for now contains just 1 repo.
I have also try to modify the gitaly timeout(from interface), modify the instance size(all nodes), modify the workers number on gitaly instace, but no fortune. Any hint?
GitLab 14.6.0
GitLab Shell 13.22.1
GitLab Workhorse v14.6.0
GitLab API v4
GitLab Pages 1.49.0
Ruby 2.7.5p203
Rails 6.1.4.1
PostgreSQL 12.8
Redis 5.0.6
From praefect i can read this error:
{“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“grpc.code”:“OK”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_type”:“bidi_stream”,“grpc.method”:“HasLocalBranches”,“grpc.request.deadline”:“2022-01-09T10:47:17.035”,“grpc.request.fullMethod”:"/gitaly.RepositoryService/HasLocalBranches",“grpc.service”:“gitaly.RepositoryService”,“grpc.start_time”:“2022-01-09T10:47:02.335”,“grpc.time_ms”:3.455,“level”:“info”,“msg”:“finished streaming call with code OK”,“peer.address”:“10.10.14.210:41034”,“pid”:3002,“relative_path”:"@hashed/c2/35/c2356069e9d1e79ca924378153cfbbfb4d4416b1f99d41a2940bfdb66c5319db.git",“remote_ip”:“37.182.113.15”,“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:02.338Z”,“username”:“gianmarco.carrieri”,“virtual_storage”:“default”} {“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“error”:“rpc error: code = NotFound desc = not found: .gitattributes”,“grpc.code”:“NotFound”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_type”:“bidi_stream”,“grpc.method”:“TreeEntry”,“grpc.request.deadline”:“2022-01-09T10:47:32.075”,“grpc.request.fullMethod”:"/gitaly.CommitService/TreeEntry",“grpc.service”:“gitaly.CommitService”,“grpc.start_time”:“2022-01-09T10:47:02.375”,“grpc.time_ms”:4.404,“level”:“info”,“msg”:“finished streaming call with code NotFound”,“peer.address”:“10.10.14.210:41034”,“pid”:3002,“relative_path”:"@hashed/c2/35/c2356069e9d1e79ca924378153cfbbfb4d4416b1f99d41a2940bfdb66c5319db.git",“remote_ip”:“37.182.113.15”,“sentry.skip”:{},“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:02.380Z”,“username”:“gianmarco.carrieri”,“virtual_storage”:“default”} {“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“error”:“rpc error: code = Canceled desc = context canceled”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_type”:“bidi_stream”,“grpc.method”:“UserCommitFiles”,“grpc.request.deadline”:“2022-01-09T10:47:59.082”,“grpc.request.fullMethod”:"/gitaly.OperationService/UserCommitFiles",“grpc.service”:“gitaly.OperationService”,“grpc.start_time”:“2022-01-09T10:47:02.382”,“level”:“error”,“msg”:“proxying to secondary failed”,“peer.address”:“10.10.14.210:41034”,“pid”:3002,“relative_path”:"@hashed/c2/35/c2356069e9d1e79ca924378153cfbbfb4d4416b1f99d41a2940bfdb66c5319db.git",“remote_ip”:“37.182.113.15”,“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:59.003Z”,“username”:“gianmarco.carrieri”,“virtual_storage”:“default”} {“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“error”:“rpc error: code = Canceled desc = context canceled”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_type”:“bidi_stream”,“grpc.method”:“UserCommitFiles”,“grpc.request.deadline”:“2022-01-09T10:47:59.082”,“grpc.request.fullMethod”:"/gitaly.OperationService/UserCommitFiles",“grpc.service”:“gitaly.OperationService”,“grpc.start_time”:“2022-01-09T10:47:02.382”,“level”:“error”,“msg”:“proxying to secondary failed”,“peer.address”:“10.10.14.210:41034”,“pid”:3002,“relative_path”:"@hashed/c2/35/c2356069e9d1e79ca924378153cfbbfb4d4416b1f99d41a2940bfdb66c5319db.git",“remote_ip”:“37.182.113.15”,“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:59.003Z”,“username”:“gianmarco.carrieri”,“virtual_storage”:“default”} {“component”:“transactions.Manager”,“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_type”:“bidi_stream”,“grpc.method”:“UserCommitFiles”,“grpc.request.deadline”:“2022-01-09T10:47:59.082”,“grpc.request.fullMethod”:"/gitaly.OperationService/UserCommitFiles",“grpc.service”:“gitaly.OperationService”,“grpc.start_time”:“2022-01-09T10:47:02.382”,“level”:“info”,“msg”:“transaction completed”,“peer.address”:“10.10.14.210:41034”,“pid”:3002,“relative_path”:"@hashed/c2/35/c2356069e9d1e79ca924378153cfbbfb4d4416b1f99d41a2940bfdb66c5319db.git",“remote_ip”:“37.182.113.15”,“sentry.skip”:{},“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:59.003Z”,“transaction.committed”:“0/3”,“transaction.id”:11,“transaction.subtransactions”:1,“username”:“gianmarco.carrieri”,“virtual_storage”:“default”} {“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“error”:“rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled”,“grpc.code”:“Canceled”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_type”:“bidi_stream”,“grpc.method”:“UserCommitFiles”,“grpc.request.deadline”:“2022-01-09T10:47:59.082”,“grpc.request.fullMethod”:"/gitaly.OperationService/UserCommitFiles",“grpc.service”:“gitaly.OperationService”,“grpc.start_time”:“2022-01-09T10:47:02.382”,“grpc.time_ms”:56621.3,“level”:“info”,“msg”:“finished streaming call with code Canceled”,“peer.address”:“10.10.14.210:41034”,“pid”:3002,“relative_path”:"@hashed/c2/35/c2356069e9d1e79ca924378153cfbbfb4d4416b1f99d41a2940bfdb66c5319db.git",“remote_ip”:“37.182.113.15”,“sentry.skip”:{},“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:59.003Z”,“username”:“gianmarco.carrieri”,“virtual_storage”:“default”} {“component”:“transactions.Manager”,“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“error”:“transaction has been canceled”,“grpc.meta.deadline_type”:“unknown”,“grpc.meta.method_type”:“unary”,“grpc.method”:“VoteTransaction”,“grpc.request.deadline”:“2022-01-09T10:47:59.082”,“grpc.request.fullMethod”:"/gitaly.RefTransaction/VoteTransaction",“grpc.request.repo”:null,“grpc.service”:“gitaly.RefTransaction”,“grpc.start_time”:“2022-01-09T10:47:02.433”,“level”:“error”,“msg”:“VoteTransaction: transaction was canceled”,“peer.address”:“10.10.14.250:8075”,“pid”:3002,“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:59.003Z”,“transaction.hash”:“5f5adbe4cce12c2cf7d1eb5381c6f9bc869bf857”,“transaction.id”:11,“transaction.voter”:“gitaly-2”} {“correlation_id”:“01FRZ75Q9QP4NDDXXQV08MJ696”,“error”:“rpc error: code = Canceled desc = transaction has been canceled”,“grpc.code”:“Canceled”,“grpc.meta.deadline_type”:“unknown”,“grpc.meta.method_type”:“unary”,“grpc.method”:“VoteTransaction”,“grpc.request.deadline”:“2022-01-09T10:47:59.082”,“grpc.request.fullMethod”:"/gitaly.RefTransaction/VoteTransaction",“grpc.request.repo”:null,“grpc.service”:“gitaly.RefTransaction”,“grpc.start_time”:“2022-01-09T10:47:02.433”,“grpc.time_ms”:56570.742,“level”:“info”,“msg”:“finished unary call with code Canceled”,“peer.address”:“10.10.14.250:8075”,“pid”:3002,“span.kind”:“server”,“system”:“grpc”,“time”:“2022-01-09T10:47:59.003Z”}