Gitlab-lfs behaves differently from gitlab-uploads: NoSuchBucket error with Digital Ocean Spaces

How to Use GitLab Self-managed
1

Additional details are available in this issue and below: #421675

As a quick summary: Consolidated object storage configuration works for gitlab-uploads but not for gitlab-lfs. It seems to be that there might be inconsistency with how the consolidated configuration is interpreted between the two.

I am wondering if any other users have experienced this?

More details from the linked issue

Summary

Using Digital Ocean spaces via the consolidated form works for gitlab-uploads (avatars, etc) but not gitlab-lfs objects. This is from a fresh install.

Note that this is distinct from the following: After upgrading Gitlab from 14.10 to 15.0.2 artifact uploading to S3 compatible object store fails: NoSuchBucket (#364814) · Issues · GitLab.org / GitLab · GitLab

Steps to reproduce

Using the following object_store configuration:

gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'region' => 'us-east-1',
  'aws_access_key_id' => '<snip>',
  'aws_secret_access_key' => '<snip>',
  'endpoint' => 'https://<snip>.<snip>.digitaloceanspaces.com',
  'path_style' => true
}

Objects like user avatars are uploaded and served correctly. And the gitlab-uploads folder is created automatically in the store.

With the same configuration pushing git-lfs files gives:

Expected(200) <=> Actual(404 Not Found)
excon.error.response
    :body => "<?xml version="1.0" encoding="UTF-8"?>
                <Error><Code>NoSuchBucket</Code>
                <BucketName><snip></BucketName>
                <RequestId><snip></RequestId>
                <HostId><snip></HostId>
            </Error>"
    :cookies           => []
    :headers           => {
        "accept-ranges"                          => "bytes"
        "cache-control"                          => "max-age=60"
        "content-length"                         => "224"
        "content-type"                           => "application/xml"
        "date"                                   => "Fri, 11 Aug 2023 00:43:26 GMT"
        "strict-transport-security"              => "max-age=15552000; includeSubDomains; preload"
        "x-amz-request-id"                       => "<snip>"
        "x-envoy-upstream-healthchecked-cluster" => ""
    }
    :host              => "<snip>"
    :local_address     => "<snip>"
    :local_port        => 55590
    :method            => "PUT"
    :omit_default_port => false
    :path              => "/gitlab-lfs/b5/fa/56f2f5fea81ffb12b6997c97fe9415c0daff9825b1ab2df7ca3f00dba8af"
    :port              => 443
    :query             => nil
    :reason_phrase     => "Not Found"
    :remote_ip         => "<snip>"
    :scheme            => "https"
    :status            => 404
    :status_line       => "HTTP/1.1 404 Not Found"

If I switch off 'path_style' => true then gitlab-uploads no longer work and gitlab-lfs still doesn’t go through. Though with a different error.

I notice that bucket names are set differently between the two. And am wondering if there is a difference in how these names are formatted.