I wanted to do some more thorough logging. Here are the results:
- tail -f /var/log/gitlab/gitlab-rails/production.log
- tail -f /var/log/gitlab/gitlab-rails/production_json.log
- tail -f /var/log/gitlab/gitlab-rails/api_json.log
- And in case that isn’t enough, I’ve tried to log as much of gitlab-ctl tail right when I click the Create merge request button: sudo gitlab-ctl tail
(They’re long, so I don’t want to copy/paste the results on this Topic)
Obviously I gave generic names for the Team, Project, IPs and our Website.
The most important piece, I think, looks to be in production.log:
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"POST","path":"/team/project/-/merge_requests","format":"html","controller":"Projects::MergeRequests::CreationsController","action":"create","status":500,"time":"2021-08-03T16:56:23.192Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"merge_request","value":{"title":"[FILTERED]","description":"[FILTERED]","assignee_ids":["0"],"reviewer_ids":["0"],"label_ids":[""],"force_remove_source_branch":"0","squash":"0","lock_version":"0","source_project_id":"41","source_branch":"AscentCharacterAttributes","target_project_id":"41","target_branch":"master"}},{"key":"merge_request_diff_head_sha","value":"f9d4a01e30d74b8b843e7b10b2810795e5c8966b"},{"key":"namespace_id","value":"team"},{"key":"project_id","value":"project"}],"remote_ip":"1.2.3.4","user_id":4,"username":"names_are_useless","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0","correlation_id":"01FC6F15DTJ7VQD0WE820Z2ZW0","meta.user":"names_are_useless","meta.project":"team/project","meta.root_namespace":"team","meta.caller_id":"Projects::MergeRequests::CreationsController#create","meta.remote_ip":"1.2.3.4","meta.feature_category":"code_review","meta.client_id":"user/4","gitaly_calls":8,"gitaly_duration_s":82.237132,"redis_calls":13,"redis_duration_s":0.00787,"redis_read_bytes":2516,"redis_write_bytes":1470,"redis_cache_calls":12,"redis_cache_duration_s":0.007299,"redis_cache_read_bytes":2335,"redis_cache_write_bytes":722,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000571,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":748,"db_count":29,"db_write_count":9,"db_cached_count":4,"cpu_s":2.260525,"mem_objects":250013,"mem_bytes":67239344,"mem_mallocs":75906,"mem_total_bytes":77239864,"exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::UnableToSend: no connection to the server\n","exception.backtrace":["app/services/issuable_base_service.rb:209:in `create'","app/services/merge_requests/base_service.rb:105:in `create'","app/services/merge_requests/create_service.rb:29:in `create'","app/services/merge_requests/create_service.rb:13:in `execute'","app/controllers/projects/merge_requests/creations_controller.rb:22:in `create'","app/controllers/application_controller.rb:483:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:474:in `set_session_storage'","lib/gitlab/i18n.rb:99:in `with_locale'","lib/gitlab/i18n.rb:105:in `with_user_locale'","app/controllers/application_controller.rb:468:in `set_locale'","app/controllers/application_controller.rb:462:in `set_current_context'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.28854,"view_duration_s":0.0,"duration_s":84.33137}
==> /var/log/gitlab/gitlab-rails/production.log <==
Completed 500 Internal Server Error in 85422ms (ActiveRecord: 164.6ms | Elasticsearch: 0.0ms | Allocations: 232093)
ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server):
What causes the ActiveRecord::StatementInvalid (PG::UnableToSend: no connection to the server ): I have no earthly idea.