Gitlab version : 14.4.0
While creating a merge request we faced the following problem where we were not able to click the merge button and it was loading forever.
Later we checked logs on one of our sidekiq and found the following issues.
400 Bad Request\\\\r\\\\n\\\"\\n\",\"error_class\":\"Excon::Error::BadRequest\",\"failed_at\":1642515019.2216873,\"retry_count\":1,\"retried_at\":1642515038.2599885}"}
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] 2022-01-18T14:11:09.298Z pid=24 tid=1ai95s WARN: Excon::Error::BadRequest: Expected(200) <=> Actual(400 Bad Request)
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] excon.error.response
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :body => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'eu-west-1' is wrong; expecting 'ap-south-1'</Message><Region>ap-south-1</Region><RequestId>00SN5V2H5X1568W6</RequestId><HostId>DDO43P/ZNK1ODaiUgEblp/kGq7EG7v3un34hmKJgfolrJhzZm/1TMPMBVOtxTOU6F1nCFhfhUZ0=</HostId></Error>"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :cookies => [
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] ]
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :headers => {
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] "Connection" => "close"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] "Content-Type" => "application/xml"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] "Date" => "Tue, 18 Jan 2022 14:11:08 GMT"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] "Server" => "AmazonS3"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] "x-amz-id-2" => "xxxxxx"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] "x-amz-request-id" => "xxxxxxx"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] }
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :host => "xxxxxx-gitlab-diff.s3.ap-south-1.amazonaws.com"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :local_address => "x.x.x.x"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :local_port => 41666
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :path => "/merge_request_diffs/mr-42303/diff-37093"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :port => 443
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :reason_phrase => "Bad Request"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :remote_ip => "x.x.x.x"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :status => 400
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq] :status_line => "HTTP/1.1 400 Bad Request\r\n"
[gitlab-sidekiq-all-in-1-v1-9579b655d-zpqcp sidekiq]
None of our kubernetes secrets being used for S3 bucket access were modified and still we don’t know from where it was taking this ‘eu-west-1’ region.