S3-Compatible Storage

Write a dedicated profile so your existing AWS credentials stay untouched:

aws configure set aws_access_key_id AKADIVMECTKQJRCUCQ2DJ5KU4VCJIQXGK6DBNVYGYZI --profile anondrop
aws configure set aws_secret_access_key wJalrXUtnFEMI-K7MDENG-bPxRfiCYEXAMPLEKEY --profile anondrop
aws configure set region us-east-1 --profile anondrop
aws configure set s3.addressing_style path --profile anondrop

Then pass --endpoint-url on each call:

export AWS_PROFILE=anondrop
export AWS_ENDPOINT_URL=https://anondrop.net/s3

aws s3 mb s3://my-bucket
aws s3 cp ./release.tar.gz s3://my-bucket/builds/release.tar.gz
aws s3 sync ./site s3://my-bucket/site --delete
aws s3 ls s3://my-bucket/builds/
aws s3 cp s3://my-bucket/builds/release.tar.gz ./release.tar.gz

If your AWS CLI is older than version 2.13 it will not read AWS_ENDPOINT_URL; pass the flag explicitly instead:

aws --profile anondrop --endpoint-url https://anondrop.net/s3 \
    s3 sync ./site s3://my-bucket/site --delete

The low-level s3api commands work too, which is how you reach headers and multipart controls directly:

aws --profile anondrop --endpoint-url https://anondrop.net/s3 \
    s3api list-objects-v2 --bucket my-bucket --prefix builds/ --delimiter /

aws --profile anondrop --endpoint-url https://anondrop.net/s3 \
    s3api put-object --bucket my-bucket --key builds/notes.txt \
    --body notes.txt --content-type text/plain

Construct the client with endpoint_url and path addressing:

import boto3
from botocore.client import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://anondrop.net/s3",
    aws_access_key_id="AKADIVMECTKQJRCUCQ2DJ5KU4VCJIQXGK6DBNVYGYZI",
    aws_secret_access_key="wJalrXUtnFEMI-K7MDENG-bPxRfiCYEXAMPLEKEY",
    region_name="us-east-1",
    config=Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},
    ),
)

s3.create_bucket(Bucket="my-bucket")

with open("release.tar.gz", "rb") as fh:
    s3.put_object(
        Bucket="my-bucket",
        Key="builds/release.tar.gz",
        Body=fh,
        ContentType="application/gzip",
        Metadata={"build": "1841", "branch": "main"},
    )

obj = s3.get_object(Bucket="my-bucket", Key="builds/release.tar.gz")
data = obj["Body"].read()
print(obj["ContentLength"], obj["ETag"], obj["Metadata"])

Ranged reads, listing with pagination and presigned links all behave the way you expect:

head = s3.get_object(
    Bucket="my-bucket", Key="builds/release.tar.gz", Range="bytes=0-1023")
first_kib = head["Body"].read()

paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="my-bucket", Prefix="builds/"):
    for item in page.get("Contents", []):
        print(item["Key"], item["Size"])

link = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "builds/release.tar.gz"},
    ExpiresIn=3600,
)
print(link)

Add this block to your rclone.conf, or answer the equivalent prompts under rclone config:

[anondrop]
type = s3
provider = Other
env_auth = false
access_key_id = AKADIVMECTKQJRCUCQ2DJ5KU4VCJIQXGK6DBNVYGYZI
secret_access_key = wJalrXUtnFEMI-K7MDENG-bPxRfiCYEXAMPLEKEY
region = us-east-1
endpoint = https://anondrop.net/s3
location_constraint =
force_path_style = true
acl = private
chunk_size = 64M
upload_concurrency = 4

Then use it like any other remote:

rclone mkdir anondrop:my-bucket
rclone copy ./photos anondrop:my-bucket/photos --progress
rclone sync ./site anondrop:my-bucket/site --progress
rclone lsl anondrop:my-bucket/photos
rclone cat anondrop:my-bucket/photos/list.txt

Put this in ~/.s3cfg. The bucket sits in the path, which is what keeps s3cmd in path-style mode:

[default]
access_key = AKADIVMECTKQJRCUCQ2DJ5KU4VCJIQXGK6DBNVYGYZI
secret_key = wJalrXUtnFEMI-K7MDENG-bPxRfiCYEXAMPLEKEY
host_base = anondrop.net/s3
host_bucket = anondrop.net/s3/%(bucket)s
bucket_location = us-east-1
use_https = True
signature_v2 = False
check_ssl_certificate = True
multipart_chunk_size_mb = 64

s3cmd mb s3://my-bucket
s3cmd put release.tar.gz s3://my-bucket/builds/release.tar.gz
s3cmd sync ./site/ s3://my-bucket/site/
s3cmd ls s3://my-bucket/builds/
s3cmd get s3://my-bucket/builds/release.tar.gz

A presigned URL carries the signature in the query string, so anyone holding the link can use it without a key of their own until it expires. The link inherits the permissions and bucket scoping of the key that signed it, and dies the moment that key is revoked.

aws --profile anondrop --endpoint-url https://anondrop.net/s3 \
    s3 presign s3://my-bucket/builds/release.tar.gz --expires-in 3600

Bucket ACLs, bucket policies, lifecycle rules, object tagging, CORS configuration, replication, encryption configuration, object lock, static website hosting, access logging, notification configuration, transfer acceleration, requester pays, analytics, inventory, metrics, public access block and ownership controls are all answered with NotImplemented. Versioning is not offered either; the versioning subresource answers with an empty configuration so clients that probe it keep working.

The metadata spelling is the one the AWS CLI can reach without any extra code:

aws --profile anondrop --endpoint-url https://anondrop.net/s3 \
    s3api put-object --bucket my-bucket --key drops/build.zip \
    --body build.zip --metadata anondrop-expires=86400

In boto3 the headers go on before the request is signed, which means they are covered by the signature:

def anondrop_headers(request, **kwargs):
    request.headers.add_header("x-anondrop-expires", "3600")
    request.headers.add_header("x-anondrop-max-downloads", "5")
    request.headers.add_header("x-anondrop-password", "correct-horse-battery")

s3.meta.events.register("before-sign.s3.PutObject", anondrop_headers)

with open("build.zip", "rb") as fh:
    s3.put_object(Bucket="my-bucket", Key="drops/build.zip", Body=fh)

s3.meta.events.unregister("before-sign.s3.PutObject", anondrop_headers)

With curl against a presigned PUT the headers are simply extra headers on the request:

curl -X PUT --upload-file build.zip \
     -H "x-anondrop-expires: 86400" \
     -H "x-anondrop-max-downloads: 5" \
     "$PRESIGNED_PUT_URL"

• x-amz-meta-* is stored verbatim and returned on GetObject and HeadObject.
• Content-Type is stored and replayed; objects uploaded without one are served as application/octet-stream.
• Content-MD5 is checked against the received bytes and a mismatch is rejected with BadDigest, with nothing written.
• The ETag of a single-shot upload is the MD5 of the bytes stored. A multipart object uses the S3 composite form instead, the MD5 of the concatenated part digests followed by a dash and the part count.
• Every object reports storage class STANDARD.

• 1000 keys per listing page, and 1000 is also the ceiling for max-keys.
• 1000 keys per DeleteObjects request.
• Part numbers 1 to 10000, and every part except the last must be at least 5 MB.
• 7 days is the longest a presigned URL may be valid for.
• 15 minutes of clock skew is tolerated; past that a request is refused with RequestTimeTooSkewed, so keep NTP running.
• Bucket names are 3 to 63 characters, lowercase letters, digits, dots and hyphens, must start and end alphanumerically, and may not look like an IPv4 address.

Going over the request ceiling is answered with SlowDown and going over a storage or transfer allowance with QuotaExceeded; both clear on their own, the first at the top of the next minute and the second when you free space or the month rolls over. Every S3 client already retries SlowDown with backoff.

Quote the RequestId if you ever need to ask us about a request. It is also returned in the x-amz-request-id response header.

A presigned URL whose X-Amz- query parameters are missing, malformed or out of range is refused with AuthorizationQueryParametersError and status 400 before any of the codes below are reached.