500 Error when trying to Create Merge Requests

I run my own Private GitLab CE and I’m updated to the latest version of 14.1.2.

As the title says: when I (or anyone else) clicks the Create merge request, we receive the 500 - Whoops, something went wrong on our end. This isn’t even a large Branch: about 6 files with a few lines of code each.

Here are some logs when I try creating a Merge Request:

root@gitlab:~# sudo gitlab-ctl tail | grep error
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
2021-08-02_01:32:01.06995 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-02T01:32:01.069922Z","queryType":"random_walk_with_error"}
2021-08-02_01:32:01.07003 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-02T01:32:01.069948Z","queryType":"server_error_500"}
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-02T18:09:19.919Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"23.82.194.104","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0","correlation_id":"01FC40SVDY9T1DQ0YTY1FMN7NM","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"23.82.194.104","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":87.627672,"redis_calls":15,"redis_duration_s":0.00976,"redis_read_bytes":2886,"redis_write_bytes":2254,"redis_cache_calls":14,"redis_cache_duration_s":0.007988,"redis_cache_read_bytes":2705,"redis_cache_write_bytes":859,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001772,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":1395,"db_count":31,"db_write_count":9,"db_cached_count":4,"cpu_s":2.259331,"mem_objects":246072,"mem_bytes":63989931,"mem_mallocs":58706,"mem_total_bytes":73832811,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.20229,"view_duration_s":0.0,"duration_s":89.59493}
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
{"args":["/opt/gitlab/embedded/bin/git","--git-dir","/var/opt/gitlab/git-data/repositories/@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","-c","core.fsyncObjectFiles=true","-c","gc.auto=0","-c","core.autocrlf=input","log","--pretty=%H","--reverse","--max-count","2147483647","master..0000000000000000000000000000000000000000","--end-of-options"],"command.exitCode":128,"command.inblock":0,"command.maxrss":63964,"command.oublock":0,"command.real_time_ms":6.851257,"command.system_time_ms":6.352,"command.user_time_ms":0,"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"error","msg":"fatal: Invalid revision range master..0000000000000000000000000000000000000000\n","path":"/opt/gitlab/embedded/bin/git","peer.address":"@","pid":900,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","error":"exit status 128","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"CommitsBetween","grpc.request.deadline":"2021-08-02T18:13:20.661Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"team/project","grpc.request.glRepository":"project-41","grpc.request.repoPath":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.CommitService","grpc.start_time":"2021-08-02T18:09:32.661Z","level":"info","msg":"ignoring git-log error","peer.address":"@","pid":1430,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021-08-02T18:09:32.669Z","username":"NicholuasP"}
{"severity":"WARN","time":"2021-08-02T18:09:34.054Z","class":"Git::BranchHooksService","correlation_id":"01FC40WYVZ9BBNJHH5PYVE4NWM","project_id":41,"project_path":"team/project","message":"Error creating pipeline","errors":"Reference not found","pipeline_params":{"before":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b","after":"0000000000000000000000000000000000000000","ref":"refs/heads/AscentCharacterAttributes","variables_attributes":[],"checkout_sha":null},"retry":0}
root@gitlab:~# sudo gitlab-ctl tail nginx/gitlab_error.log
(Nothing)
root@gitlab:~# tail -f /var/log/gitlab/gitlab-rails/production.log | grep error
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
root@gitlab:~# tail -f /var/log/gitlab/nginx/error.log
(Nothing)
root@gitlab:~# tail -f /var/log/gitlab/nginx/gitlab_error.log
(Nothing)

Now I’ve been having on-going issues with the 500 Error; this includes Merging and Committing as well. As soon as I believe I’ve fixed the issue, a new issue rears its ugly head.

Any ideas?

I wanted to do some more thorough logging. Here are the results:

(They’re long, so I don’t want to copy/paste the results on this Topic)

Obviously I gave generic names for the Team, Project, IPs and our Website.


The most important piece, I think, looks to be in production.log:

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-03T16:56:23.192Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"1.2.3.4","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0","correlation_id":"01FC6F15DTJ7VQD0WE820Z2ZW0","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"1.2.3.4","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":82.237132,"redis_calls":13,"redis_duration_s":0.00787,"redis_read_bytes":2516,"redis_write_bytes":1470,"redis_cache_calls":12,"redis_cache_duration_s":0.007299,"redis_cache_read_bytes":2335,"redis_cache_write_bytes":722,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000571,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":748,"db_count":29,"db_write_count":9,"db_cached_count":4,"cpu_s":2.260525,"mem_objects":250013,"mem_bytes":67239344,"mem_mallocs":75906,"mem_total_bytes":77239864,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.28854,"view_duration_s":0.0,"duration_s":84.33137}

==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 500 Internal Server Error in 85422ms (ActiveRecord: 164.6ms | Elasticsearch: 0.0ms | Allocations: 232093)
ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server):

What causes the ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server ): I have no earthly idea.

Getting the 500 error again with another Merge Request (this one is 11 Commits with 397 Changes), but this time production.log is a bit different:

root@gitlab:~# tail -f /var/log/gitlab/gitlab-rails/production.log
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 2ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 85)
Completed 200 OK in 4ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 85)
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 3ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 85)
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 2ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 85)
Started POST "/team/project/-/merge_requests" for 1.2.3.4 at 2021-08-16 17:07:14 +0000
Processing by Projects::MergeRequests::CreationsController#create as HTML
  Parameters: {"authenticity_token"=>"[FILTERED]", "merge_request"=>{"title"=>"[FILTERED]", "description"=>"[FILTERED]", "assignee_ids"=>["0"], "reviewer_ids"=>["0"], "label_ids"=>[""], "force_remove_source_branch"=>"0", "squash"=>"0", "lock_version"=>"0", "source_project_id"=>"41", "source_branch"=>"LuosCaves_Relocate", "target_project_id"=>"41", "target_branch"=>"master"}, "merge_request_diff_head_sha"=>"41d3e1bf4525fb8be48c66f610b0fe58ae9b9718", "namespace_id"=>"team", "project_id"=>"project"}

Completed 500 Internal Server Error in 113926ms (ActiveRecord: 244.3ms | Elasticsearch: 0.0ms | Allocations: 36527)

ActiveRecord::StatementInvalid (PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
):

app/services/issuable_base_service.rb:209:in `create'
app/services/merge_requests/base_service.rb:105:in `create'
app/services/merge_requests/create_service.rb:29:in `create'
app/services/merge_requests/create_service.rb:13:in `execute'
app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'
app/controllers/application_controller.rb:483:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:474:in `set_session_storage'
lib/gitlab/i18n.rb:99:in `with_locale'
lib/gitlab/i18n.rb:105:in `with_user_locale'
app/controllers/application_controller.rb:468:in `set_locale'
app/controllers/application_controller.rb:462:in `set_current_context'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

I have since added the following to my /etc/gitlab/gitlab.rb file:

postgresql['idle_in_transaction_session_timeout'] = "360000" # I receive the 500 error in less than 6 minutes, so this clearly isn’t helping
gitlab_rails['db_max_allowed_packet'] = "20G" # The combined Merge Request is far less than 20 GB, probably a couple GB, so this clearly isn’t helping

I now receive the following:

root@gitlab:~# tail -f /var/log/gitlab/gitlab-rails/production.log
Started POST "/api/graphql" for 174.130.57.241 at 2021-08-16 17:52:08 +0000
Completed 200 OK in 1218ms (Views: 2.0ms | ActiveRecord: 41.6ms | Elasticsearch: 0.0ms | Allocations: 37585)
Started GET "/uploads/-/system/user/avatar/4/avatar.png?width=40" for 174.130.57.241 at 2021-08-16 17:52:09 +0000
Processing by UploadsController#show as HTML
  Parameters: {"width"=>"40", "model"=>"user", "mounted_as"=>"avatar", "id"=>"4", "filename"=>"avatar.png"}
Completed 200 OK in 26ms (ActiveRecord: 4.0ms | Elasticsearch: 0.0ms | Allocations: 3603)
Completed 200 OK in 1281ms (Views: 1.7ms | ActiveRecord: 7.9ms | Elasticsearch: 0.0ms | Allocations: 30180)
Processing by GraphqlController#execute as */*
  Parameters: {"_json"=>[{"operationName"=>"getPermissions", "variables"=>"[FILTERED]", "query"=>"query getPermissions($projectPath: ID!) {\n  project(fullPath: $projectPath) {\n    __typename\n    userPermissions {\n      __typename\n      pushCode\n      forkProject\n      createMergeRequestIn\n    }\n  }\n}\n"}, {"operationName"=>"getFiles", "variables"=>"[FILTERED]", "query"=>"fragment PageInfo on PageInfo {\n  __typename\n  hasNextPage\n  hasPreviousPage\n  startCursor\n  endCursor\n}\n\nfragment TreeEntry on Entry {\n  __typename\n  id\n  sha\n  name\n  flatPath\n  type\n}\n\nquery getFiles($projectPath: ID!, $path: String, $ref: String!, $pageSize: Int!, $nextPageCursor: String) {\n  project(fullPath: $projectPath) {\n    __typename\n    repository {\n      __typename\n      tree(path: $path, ref: $ref) {\n        __typename\n        trees(first: $pageSize, after: $nextPageCursor) {\n          __typename\n          edges {\n            __typename\n            node {\n              ...TreeEntry\n              webPath\n              __typename\n            }\n          }\n          pageInfo {\n            ...PageInfo\n            __typename\n          }\n        }\n        submodules(first: $pageSize, after: $nextPageCursor) {\n          __typename\n          edges {\n            __typename\n            node {\n              ...TreeEntry\n              webUrl\n              treeUrl\n              __typename\n            }\n          }\n          pageInfo {\n            ...PageInfo\n            __typename\n          }\n        }\n        blobs(first: $pageSize, after: $nextPageCursor) {\n          __typename\n          edges {\n            __typename\n            node {\n              ...TreeEntry\n              mode\n              webPath\n              lfsOid\n              __typename\n            }\n          }\n          pageInfo {\n            ...PageInfo\n            __typename\n          }\n        }\n      }\n    }\n  }\n}\n"}], "graphql"=>{"_json"=>[{"operationName"=>"getPermissions", "variables"=>"[FILTERED]", "query"=>"query getPermissions($projectPath: ID!) {\n  project(fullPath: $projectPath) {\n    __typename\n    userPermissions {\n      __typename\n      pushCode\n      forkProject\n      createMergeRequestIn\n    }\n  }\n}\n"}, {"operationName"=>"getFiles", "variables"=>"[FILTERED]", "query"=>"fragment PageInfo on PageInfo {\n  __typename\n  hasNextPage\n  hasPreviousPage\n  startCursor\n  endCursor\n}\n\nfragment TreeEntry on Entry {\n  __typename\n  id\n  sha\n  name\n  flatPath\n  type\n}\n\nquery getFiles($projectPath: ID!, $path: String, $ref: String!, $pageSize: Int!, $nextPageCursor: String) {\n  project(fullPath: $projectPath) {\n    __typename\n    repository {\n      __typename\n      tree(path: $path, ref: $ref) {\n        __typename\n        trees(first: $pageSize, after: $nextPageCursor) {\n          __typename\n          edges {\n            __typename\n            node {\n              ...TreeEntry\n              webPath\n              __typename\n            }\n          }\n          pageInfo {\n            ...PageInfo\n            __typename\n          }\n        }\n        submodules(first: $pageSize, after: $nextPageCursor) {\n          __typename\n          edges {\n            __typename\n            node {\n              ...TreeEntry\n              webUrl\n              treeUrl\n              __typename\n            }\n          }\n          pageInfo {\n            ...PageInfo\n            __typename\n          }\n        }\n        blobs(first: $pageSize, after: $nextPageCursor) {\n          __typename\n          edges {\n            __typename\n            node {\n              ...TreeEntry\n              mode\n              webPath\n              lfsOid\n              __typename\n            }\n          }\n          pageInfo {\n            ...PageInfo\n            __typename\n          }\n        }\n      }\n    }\n  }\n}\n"}]}}
Completed 200 OK in 2402ms (Views: 19.6ms | ActiveRecord: 54.5ms | Elasticsearch: 0.0ms | Allocations: 22120)
Started POST "/team/project/-/merge_requests" for 50.246.69.13 at 2021-08-16 17:59:18 +0000
Processing by Projects::MergeRequests::CreationsController#create as HTML
  Parameters: {"authenticity_token"=>"[FILTERED]", "merge_request"=>{"title"=>"[FILTERED]", "description"=>"[FILTERED]", "assignee_ids"=>["0"], "reviewer_ids"=>["0"], "label_ids"=>[""], "force_remove_source_branch"=>"0", "squash"=>"0", "lock_version"=>"0", "source_project_id"=>"41", "source_branch"=>"LuosCaves_Relocate", "target_project_id"=>"41", "target_branch"=>"master"}, "merge_request_diff_head_sha"=>"41d3e1bf4525fb8be48c66f610b0fe58ae9b9718", "namespace_id"=>"team", "project_id"=>"project"}

Started GET "/team/project/-/merge_requests/new/diffs.json?merge_request%5Bsource_project_id%5D=41&merge_request%5Bsource_branch%5D=LuosCaves_Relocate&merge_request%5Btarget_project_id%5D=41&merge_request%5Btarget_branch%5D=master" for 50.246.69.13 at 2021-08-16 18:00:19 +0000
Processing by Projects::MergeRequests::CreationsController#diffs as JSON
  Parameters: {"merge_request"=>{"source_project_id"=>"41", "source_branch"=>"LuosCaves_Relocate", "target_project_id"=>"41", "target_branch"=>"master"}, "namespace_id"=>"team", "project_id"=>"project"}
Completed 500 Internal Server Error in 113673ms (ActiveRecord: 431.2ms | Elasticsearch: 0.0ms | Allocations: 64521)

GRPC::DeadlineExceeded (4:Deadline Exceeded. debug_error_string:{"created":"@1629136872.525562385","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}):

grpc-1.30.2-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status'
grpc-1.30.2-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
grpc-1.30.2-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:169:in `receive_and_check_status'
grpc-1.30.2-x86_64 (linux) src/ruby/lib/grpc/generic/active_call.rb:337:in `each_remote_read_then_finish'
puma: cluster worker 17: 1457 [gitlab-puma-worker]:in `each'

I have a hard time believing it would be related to the allocated CPU or Memory for my GitLab VM: Both never go above 50% before I get the 500 error page.

So, I was able to resolve this issue. I had to increase the Puma Worker Timeout and Gitaly Timeouts.

This actually goes back to a previous issue I was having where I increased the Puma Worker and Gitaly Timeouts. Only this time I was much more drastic with the values:

  • puma['worker_timeout'] = 120 TO puma['worker_timeout'] = 300 in /etc/gitlab/gitlab.rb
  • Gitaly Default Timeout Period: 114 TO 285
  • Gitaly Fast Timeout Period: 114 TO 285
  • Medium Timeout Period: 114 TO 285

CPU usage is up from around 30% to 60% when a Merge Request is being created, but Memory Usage is still about the same.


Now, I’m not completely out of the woods yet: with this Merge Request, I do have Checking if merge request can be merged… on-loop, which has been a re-occurring issue I’ve been having.

In the past I was able to just Close and then Re-open a large Merge Request to either receive the There are merge conflicts message or the green Merge button, but not for this particular Merge Request.

EDIT: Actually, I just resolved the Merge Loop error as well. See this post.