GitLab Export API to S3 returning 403

General
, , , ,
1

Same question as here: How to export a gitlab project to S3? … but no answer.

I’m attempting to use the GitLab Import/Export API to backup gitlab.com projects to an AWS S3 bucket, as outlined here: Project import and export API | GitLab Docs

The API request itself returns 202 accepted. However the upload fails with:

 Project Foo couldn't be exported.

The errors we encountered were:

    Invalid response code while uploading file. Code: 403

Here is the code for generating the presigned AWS url:

s3_client = boto3.client("s3")
url = s3_client.generate_presigned_url(
    ClientMethod="put_object",
    Params={"Bucket": "mybucket", "Key": "backups/gitlab/foo"},
    ExpiresIn=3600,
)
print(url)

The presigned url works with curl:

curl <generated-url> --upload-file somefile

It is not an expiration issue, I am able to successfully use the url (via curl) after the GitLab upload has failed.

API request url is https://gitlab.com/api/v4/projects/123456789/export.
Form data is:
upload[http_method] = PUT
upload[url] = <my presigned url>

I’ve tried the form data as form-data and x-www-form-urlencoded.

Any help is appreciated.

3

Finally solved this.

The following script is from aws docs: Uploading objects with presigned URLs - Amazon Simple Storage Service

import argparse
import boto3
from botocore.exceptions import ClientError
from botocore.client import Config

def generate_presigned_url(s3_client, client_method, method_parameters, expires_in):
    """
    Generate a presigned Amazon S3 URL that can be used to perform an action.
    
    :param s3_client: A Boto3 Amazon S3 client.
    :param client_method: The name of the client method that the URL performs.
    :param method_parameters: The parameters of the specified client method.
    :param expires_in: The number of seconds the presigned URL is valid for.
    :return: The presigned URL.
    """
    try:
        url = s3_client.generate_presigned_url(
            ClientMethod=client_method,
            Params=method_parameters,
            ExpiresIn=expires_in
        )
    except ClientError:
        print(f"Couldn't get a presigned URL for client method '{client_method}'.")
        raise
    return url

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("bucket", help="The name of the bucket.")
    parser.add_argument(
        "key", help="The key (path and filename) in the S3 bucket.",
    )
    args = parser.parse_args()
    
    # By default, this will use credentials from ~/.aws/credentials
    s3_client = boto3.client("s3", region_name="us-west-2", config=Config(signature_version="s3v4"))
    
    # The presigned URL is specified to expire in 1000 seconds
    url = generate_presigned_url(
        s3_client, 
        "put_object", 
        {"Bucket": args.bucket, "Key": args.key}, 
        1000
    )
    print(url)

if __name__ == "__main__":
    main()

The ticket is in passing the config to the boto3 client: Config(signature_version="s3v4")