How to Use S3 Presigned URLs for Secure Uploads and Downloads in Python

The S3 protocol is commonly used for object storage services, enabling upload and download functions. When a program lacks permission or needs reduced privileges for security, presigned URLs can be used to perform unauthenticated read/write operations.

PUT Upload

import boto3


def gen_s3_presigned_put(bucket: str, path: str) -> str:
    s3r = boto3.resource(
        's3',
        endpoint_url=S3_ENDPOINT,
        aws_access_key_id=S3_ACCESS_KEY,
        aws_secret_access_key=S3_SECRET_KEY,
        region_name=S3_REGION,
        config=Config(signature_version='s3v4'),
    )
    if not s3r.Bucket(bucket).creation_date:
        s3r.create_bucket(Bucket=bucket)
    return s3r.meta.client.generate_presigned_url(
        ClientMethod='put_object',
        Params={
            'Bucket': bucket,
            'Key': path,
        },
        ExpiresIn=3600,
        HttpMethod='PUT',
    )

url = generate_presigned_put('bucket', 'remote/path/of/file')

The generate_presigned_url function (source in botocore/signers.py#L245) returns a URL string that can be used for a PUT upload within one hour. Passing this URL to an unauthenticated client enables file upload.

Authentication fields such as S3_* are embedded in the URL generation but are not required when using the presigned URL. The example also includes a bucket existence check, which can be removed in production.

import requests


def upload_with_put(url):
    with open('local/path/of/file', 'rb') as file:
        response = requests.put(url, data=file)
        response.raise_for_status()

GET Download

import boto3


def gen_s3_presigned_get(bucket: str, path: str) -> str:
    s3r = boto3.resource(
        's3',
        endpoint_url=S3_ENDPOINT,
        aws_access_key_id=S3_ACCESS_KEY,
        aws_secret_access_key=S3_SECRET_KEY,
        region_name=S3_REGION,
        config=Config(signature_version='s3v4'),
    )
    if not s3r.Bucket(bucket).creation_date:
        s3r.create_bucket(Bucket=bucket)
    return s3r.meta.client.generate_presigned_url(
        ClientMethod='get_object',
        Params={
            'Bucket': bucket,
            'Key': path,
        },
        ExpiresIn=3600,
        HttpMethod='GET',
    )

url = generate_presigned_get('bucket', 'remote/path/of/file')

The GET implementation mirrors the PUT version, differing only in ClientMethod and HttpMethod.

size = 2**12  # Use 4 KB memory buffer
with open(path, 'wb') as file:
    for chunk in response.iter_content(chunk_size=size):
        file.write(chunk)

POST Upload

import boto3


def gen_s3_presigned_post(bucket: str, path: str) -> str:
    s3r = boto3.resource(
        's3',
        endpoint_url=S3_ENDPOINT,
        aws_access_key_id=S3_ACCESS_KEY,
        aws_secret_access_key=S3_SECRET_KEY,
        region_name=S3_REGION,
        config=Config(signature_version='s3v4'),
    )
    if not s3r.Bucket(bucket).creation_date:
        s3r.create_bucket(Bucket=bucket)
    dict_ = s3r.meta.client.generate_presigned_post(
        Bucket=bucket,
        Key=path,
        ExpiresIn=3600,
    )
    return dict_['url'], dict_['fields']

url, fields = generate_presigned_post('bucket', 'remote/path/of/file')

The generate_presigned_post function (source in botocore/signers.py#L605) returns a URL and a dictionary of form fields that must be submitted together with the file.

import requests


def upload_with_post(url, fields):
    with open('local/path/of/file', 'rb') as file:
        files = {'file': ('remote/path/of/file', file)}
        response = requests.post(url, data=fields, files=files)
        response.raise_for_status()

An example HTML form shows how to embed the returned url and fields as hidden inputs for a standard multipart upload.

Summary

Although both POST and PUT can be used for uploads, PUT is generally simpler and preferred because it is idempotent, while POST is not repeatable. Multipart (segmented) uploads are not supported by presigned URLs.