Skip to content

S3 API Operations

Supported S3 API operations in IOLake.

Addressing and signing

  • Path-style always; virtual-hosted where the gateway has an S3 domain. A request names its bucket in the path (https://s3.example.com/bucket/key), or, once the operator has set the gateway's S3 domain, in the host (https://bucket.s3.example.com/key): see Virtual-hosted addressing below. Path style works with any setup, so it is the safe choice for a client that doesn't know: boto3 Config(s3={"addressing_style": "path"}), the AWS CLI aws configure set default.s3.addressing_style path, the Java and Go SDKs forcePathStyle / UsePathStyle, Iceberg and Spark s3.path-style-access=true.
  • SigV4 only, for requests and presigned URLs; SigV2 is refused with 400. Against a custom endpoint, boto3 without signature_version="s3v4" and AWS CLI v1's aws s3 presign produce SigV2 presigned URLs: set Config(signature_version="s3v4") in boto3, and aws configure set default.s3.signature_version s3v4 for the CLI.
  • Overload answers 503 SlowDown (with Retry-After), when a gateway is at its memory budget or its limit of requests in flight; SDKs retry it with backoff. A request body that sends nothing for 30 s gets 400 RequestTimeout (gateway memory).
  • A write with no room answers 507 StorageFull (MinIO's XMinioStorageFull; AWS has no such error): the OSDs it needs are past their full ratio, or the filesystem under their disks or metadata is full (ENOSPC, EDQUOT). SDKs don't retry it, as they do a 503 (full disks).

Virtual-hosted addressing

The gateway reads a bucket from the host name once it is told the domains buckets are named under: --s3-domain on the gateway (repeat it, or separate domains with commas; env IOLAKE_S3_DOMAIN), --s3-domain on iolake-aio and iolake-install, and gateway.s3Domains in the Helm chart (configuration). With --s3-domain s3.example.com, a request's Host is read as S3 reads it:

Host Addressing Bucket
photos.s3.example.com virtual-hosted; the path is the key photos
my.photos.s3.example.com virtual-hosted; a bucket name may hold dots my.photos
s3.example.com path-style the path's first segment
an address, localhost, any name not under a domain path-style the path's first segment

The port is ignored and case doesn't matter. With several domains, the longest one a host is under is used (s3.example.com before example.com). Everything after works on the bucket from the host: signatures (the path and host signed are the ones sent), presigned URLs, bucket policies and IAM policies, CORS, metrics, the audit log. Bucket operations go to the bucket's own host: PUT / creates the bucket, GET /?list-type=2 lists it, POST /?delete is DeleteObjects, DELETE / deletes it. The domain itself stays path-style, so the admin API, the console, the Iceberg catalog and Delta Sharing are reached on it as before.

Two things the gateway doesn't provide, for each domain:

  • Wildcard DNS: *.s3.example.com resolving to the gateway (its load balancer or ingress), and, behind an ingress, a host rule for the wildcard (gateway.ingress.hosts, host: "*.s3.example.com").
  • A TLS certificate covering *.s3.example.com where TLS ends. A wildcard covers one label only, so a bucket name with dots (my.photos.s3.example.com) fails certificate checks over HTTPS, as on AWS: address such buckets path-style.

The domains are not taken from --external-endpoint: that would make every name under the endpoint's host a bucket, and a gateway reached at a name under it (gw1.s3.example.com) would stop working.

What's refused rather than guessed. A request whose bucket can't be told for certain gets an S3 error, never another request's answer:

  • 400 InvalidBucketName: a host under a domain whose bucket part isn't a valid bucket name (Bad_Name.s3.example.com).
  • 400 InvalidRequest: on a path-style host, a bucket operation on / — PUT /, DELETE /, HEAD /, POST /?delete, a form upload to /, or GET / with a listing's parameters (list-type, encoding-type, location, …). Only a client that put the bucket in the host sends these; read path-style they would name no bucket, and a listing would have come back as the caller's buckets. The message says to set --s3-domain (or names the domains the host isn't under), or to use path style. ListBuckets (GET /, with max-buckets, prefix, continuation-token, bucket-region), the IAM and STS query API (Action=) and CORS preflights on / are unchanged.

Without a domain the gateway can't tell bucket.s3.example.com from s3.example.com, so a request that is also valid path-style is read path-style: a bare GET / is ListBuckets, and /key is bucket key. If any client addresses buckets by host name, set --s3-domain.

Bucket Operations

ListBuckets

List all buckets owned by the authenticated user.

GET /

Response:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult>
    <Owner>
        <ID>user-id</ID>
        <DisplayName>User Name</DisplayName>
    </Owner>
    <Buckets>
        <Bucket>
            <Name>my-bucket</Name>
            <CreationDate>2024-01-15T10:30:00.000Z</CreationDate>
        </Bucket>
    </Buckets>
</ListAllMyBucketsResult>

CreateBucket

Create a new bucket.

PUT /{bucket}

Headers: - x-amz-acl: only private or bucket-owner-full-control (ACLs are owner-enforced) - x-amz-bucket-object-lock-enabled: true: a lock bucket (versioned) - x-iolake-pool: <pool>: the pool its data goes to (default: the tenant's default pool, else the cluster's default placement). A tenant may use only its default and allowed pools; the pool is fixed for the bucket's life

Response: 200 OK

A new bucket starts with public access blocked, and with no configuration from any earlier bucket of the same name.


DeleteBucket

Delete an empty bucket.

DELETE /{bucket}

Response: 204 No Content

Errors: - BucketNotEmpty: the bucket holds objects, or (versioned) noncurrent versions or delete markers

Deleting a bucket also removes its policy, object lock, lifecycle, encryption and public access settings, in the same commit.


HeadBucket

Check if a bucket exists and you have access.

HEAD /{bucket}

Response: 200 OK (bucket exists) or 404 Not Found


GetBucketLocation

Get the region of a bucket: the gateway's --region (configuration), which every bucket is in. Authorized as s3:GetBucketLocation.

GET /{bucket}?location

Response (a gateway started with --region eu-west-1):

<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">eu-west-1</LocationConstraint>

In us-east-1, the default, the constraint is empty, as S3 answers it; SDKs read an empty constraint as us-east-1:

<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>

A bucket that doesn't exist is 404 NoSuchBucket.


Object Operations

PutObject

Upload an object.

PUT /{bucket}/{key}

Headers: - Content-Type: MIME type - Content-Length: Object size - x-amz-meta-*: User metadata - Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Expires: kept with the object and served back (and kept by a copy) - If-None-Match: *: write only if no object has the key (412 PreconditionFailed if one does) — a create that can't overwrite - If-Match: <etag>: write only if the current object has this ETag (412, or 404 NoSuchKey if there's none) — compare-and-swap

Conditional writes are checked and applied atomically: of two writers racing with If-None-Match: *, exactly one wins. CompleteMultipartUpload takes the same headers.

A body may be up to 5 GiB, as S3 allows (large uploads); a larger one gets 400 EntityTooLarge before any of it is read.

Response:

HTTP/1.1 200 OK
ETag: "d41d8cd98f00b204e9800998ecf8427e"

Example:

curl -X PUT \
  -H "Content-Type: text/plain" \
  -d "Hello World" \
  http://localhost:9000/my-bucket/hello.txt

GetObject

Download an object.

GET /{bucket}/{key}

Headers: - Range: Byte range (optional)

Response Headers: - Content-Type: Object MIME type - Content-Length: Object size - ETag: Object hash - Last-Modified: Last modification time

Range Request:

curl -H "Range: bytes=0-99" http://localhost:9000/my-bucket/file.txt

The object is streamed stripe by stripe, never held whole in the gateway. A failure reading the first stripe is an error status; one after the headers have gone out closes the connection short of Content-Length (never with wrong bytes), which clients see as a short read and retry.


HeadObject

Get object metadata without body.

HEAD /{bucket}/{key}

Response Headers: - Content-Type: Object MIME type - Content-Length: Object size - ETag: Object hash - x-amz-meta-*: User metadata


DeleteObject

Delete an object.

DELETE /{bucket}/{key}

Response: 204 No Content


CopyObject

Copy an object.

PUT /{bucket}/{key}
x-amz-copy-source: /source-bucket/source-key

Response:

<?xml version="1.0" encoding="UTF-8"?>
<CopyObjectResult>
    <LastModified>2024-01-15T10:30:00.000Z</LastModified>
    <ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
</CopyObjectResult>

A copy whose source and destination are encrypted alike shares the source's stripes, at any size. A copy that changes the encryption reads the source and writes it again, streamed as a large PUT is, so its source (or, for UploadPartCopy, its range) may be at most 5 GiB, as S3 allows; a larger one gets 400 InvalidRequest, as S3 answers.


ListObjectsV2

List objects in a bucket.

GET /{bucket}?list-type=2

Parameters: - prefix: Filter by prefix - delimiter: Group by delimiter - max-keys: Max results (default: 1000) - continuation-token: Pagination token - start-after: Start listing after this key

Response:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult>
    <Name>my-bucket</Name>
    <Prefix></Prefix>
    <MaxKeys>1000</MaxKeys>
    <IsTruncated>false</IsTruncated>
    <Contents>
        <Key>file1.txt</Key>
        <LastModified>2024-01-15T10:30:00.000Z</LastModified>
        <ETag>"abc123"</ETag>
        <Size>1024</Size>
        <StorageClass>STANDARD</StorageClass>
    </Contents>
</ListBucketResult>

Multipart Upload

Multipart upload allows uploading large objects in parts. It is required for objects over 5 GiB, the most a single PUT takes, as in S3.

Limits and Behavior

Parameter Value Notes
Minimum part size 5 MiB Except the last part (400 EntityTooSmall at completion)
Maximum part size 5 GiB As S3; a larger part gets 400 EntityTooLarge before its body is read
Maximum single PUT 5 GiB As S3; a larger one gets 400 EntityTooLarge before its body is read
Maximum parts 10,000
Maximum object size 5 TiB As S3; a completion whose parts make more gets 400 EntityTooLarge
Default AWS CLI part size 8 MB multipart_chunksize in the CLI's S3 config; up to 5 GiB

Large uploads

A single PUT or an UploadPart over 16 MiB is not held in the gateway's memory: its body is streamed to the OSDs stripe by stripe as it arrives, each stripe erasure-coded (and encrypted, with SSE) and written while the next is read, a couple of stripes in flight at once. However large the upload, the gateway holds a few stripes of it (about 48 MiB with 4+2), reserved from its memory budget before the body is read: with no room, the request gets 503 SlowDown and the client retries.

The upload is acknowledged (200) only once every stripe is on stable storage at write quorum and the object's metadata is committed. The object becomes visible only then, after the whole body has arrived and passed every check it carries:

  • its signed payload hash (x-amz-content-sha256), computed over the bytes as they stream (400 XAmzContentSHA256Mismatch);
  • with aws-chunked, each chunk's signature, checked as each chunk ends (403 SignatureDoesNotMatch), and the checksum trailer;
  • Content-MD5 (400 BadDigest) and an x-amz-checksum-* header or trailer (400 BadDigest).

A body that fails a check, or a client that goes away mid-body, stores nothing: the shards already written are freed at once (as for any failed write, a shard an OSD can't be reached to free is retried for a while). A gateway that stops mid-upload leaves its shards allocated until the orphan scan on the roadmap reclaims them. An upload that meets a placement group whose acting set changed under it gets 503, which clients retry; a body small enough to be held whole is placed again without the client seeing it.

Multi-Stripe Chunking

IOLake uses a 4MB internal block size. When a part exceeds 4MB, it is automatically split into multiple stripes:

8 MB Part Upload:
┌─────────────────────────────────────────────────────┐
│                   8 MB Part Data                     │
└─────────────────────────────────────────────────────┘
                         │
              Split into ~4MB stripes
                         │
         ┌───────────────┴───────────────┐
         ▼                               ▼
    ┌─────────┐                     ┌─────────┐
    │ Stripe 0│ (4 MB)              │ Stripe 1│ (4 MB)
    └────┬────┘                     └────┬────┘
         │                               │
    EC Encode (4+2)                 EC Encode (4+2)
         │                               │
    6 shards                        6 shards

Each stripe is independently erasure-coded and distributed across storage nodes.

CreateMultipartUpload

Initiate a multipart upload.

POST /{bucket}/{key}?uploads

Response:

<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult>
    <Bucket>my-bucket</Bucket>
    <Key>large-file.zip</Key>
    <UploadId>abc123</UploadId>
</InitiateMultipartUploadResult>

UploadPart

Upload a part. Parts can be uploaded in any order and in parallel.

PUT /{bucket}/{key}?partNumber={n}&uploadId={id}

Parameters:

  • partNumber: 1 to 10,000
  • uploadId: From CreateMultipartUpload response

Response:

HTTP/1.1 200 OK
ETag: "part-etag"

Example with curl:

# Upload part 1
curl -X PUT \
  "http://localhost:9000/my-bucket/large-file?partNumber=1&uploadId=abc123" \
  --data-binary @part1.bin

CompleteMultipartUpload

Complete the upload by providing the list of parts in order.

POST /{bucket}/{key}?uploadId={id}

Request Body:

<CompleteMultipartUpload>
    <Part>
        <PartNumber>1</PartNumber>
        <ETag>"part1-etag"</ETag>
    </Part>
    <Part>
        <PartNumber>2</PartNumber>
        <ETag>"part2-etag"</ETag>
    </Part>
</CompleteMultipartUpload>

Response:

<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult>
    <Location>http://localhost:9000/my-bucket/large-file</Location>
    <Bucket>my-bucket</Bucket>
    <Key>large-file</Key>
    <ETag>"abc123-2"</ETag>
</CompleteMultipartUploadResult>

The final ETag format is "<md5-of-part-md5s>-<part-count>".


AbortMultipartUpload

Abort an in-progress upload and delete uploaded parts.

DELETE /{bucket}/{key}?uploadId={id}

Response: 204 No Content


ListParts

List uploaded parts for an in-progress multipart upload.

GET /{bucket}/{key}?uploadId={id}

Parameters:

  • max-parts: Maximum parts to return (default: 1000)
  • part-number-marker: Start listing after this part number

Response:

<?xml version="1.0" encoding="UTF-8"?>
<ListPartsResult>
    <Bucket>my-bucket</Bucket>
    <Key>large-file</Key>
    <UploadId>abc123</UploadId>
    <MaxParts>1000</MaxParts>
    <IsTruncated>false</IsTruncated>
    <Part>
        <PartNumber>1</PartNumber>
        <LastModified>2024-01-15T10:30:00.000Z</LastModified>
        <ETag>"part1-etag"</ETag>
        <Size>8388608</Size>
    </Part>
</ListPartsResult>

ListMultipartUploads

List in-progress multipart uploads for a bucket.

GET /{bucket}?uploads

Parameters:

  • prefix: Filter by key prefix
  • max-uploads: Maximum uploads to return (default: 1000)

Response:

<?xml version="1.0" encoding="UTF-8"?>
<ListMultipartUploadsResult>
    <Bucket>my-bucket</Bucket>
    <MaxUploads>1000</MaxUploads>
    <IsTruncated>false</IsTruncated>
    <Upload>
        <Key>large-file</Key>
        <UploadId>abc123</UploadId>
        <Initiated>2024-01-15T10:30:00.000Z</Initiated>
    </Upload>
</ListMultipartUploadsResult>

Example: Upload Large File with AWS CLI

# AWS CLI automatically uses multipart upload for files > 8MB
aws --endpoint-url http://localhost:9000 s3 cp large-file.zip s3://my-bucket/

# Configure part size (minimum 5MB)
aws configure set default.s3.multipart_chunksize 16MB
aws --endpoint-url http://localhost:9000 s3 cp large-file.zip s3://my-bucket/

# Manual multipart upload
aws --endpoint-url http://localhost:9000 s3api create-multipart-upload \
  --bucket my-bucket --key large-file.zip

# Upload parts
aws --endpoint-url http://localhost:9000 s3api upload-part \
  --bucket my-bucket --key large-file.zip \
  --part-number 1 --upload-id $UPLOAD_ID --body part1.bin

# Complete upload
aws --endpoint-url http://localhost:9000 s3api complete-multipart-upload \
  --bucket my-bucket --key large-file.zip \
  --upload-id $UPLOAD_ID \
  --multipart-upload file://parts.json

Bucket Policy

PutBucketPolicy

Set bucket policy.

PUT /{bucket}?policy

Request Body:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-bucket/*"
        }
    ]
}

GetBucketPolicy

Get bucket policy.

GET /{bucket}?policy

DeleteBucketPolicy

Delete bucket policy.

DELETE /{bucket}?policy

Lifecycle

PUT    /{bucket}?lifecycle
GET    /{bucket}?lifecycle      → 404 NoSuchLifecycleConfiguration when none
DELETE /{bucket}?lifecycle
<LifecycleConfiguration>
  <Rule>
    <ID>logs</ID>
    <Filter><Prefix>logs/</Prefix></Filter>
    <Status>Enabled</Status>
    <Expiration><Days>30</Days></Expiration>
  </Rule>
  <Rule>
    <ID>old-versions</ID>
    <Filter/>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>7</NoncurrentDays>
      <NewerNoncurrentVersions>3</NewerNoncurrentVersions>
    </NoncurrentVersionExpiration>
    <AbortIncompleteMultipartUpload><DaysAfterInitiation>2</DaysAfterInitiation></AbortIncompleteMultipartUpload>
  </Rule>
</LifecycleConfiguration>
action what happens
Expiration Days / Date The current version expires. Unversioned: deleted. Versioned or suspended: a delete marker is added, as a DELETE without a version id would — the data stays as a noncurrent version.
Expiration ExpiredObjectDeleteMarker A delete marker that is the key's only version is removed. Days does this too in a versioned bucket. A marker with versions behind it is never removed: that would bring the object back.
NoncurrentVersionExpiration A noncurrent version is deleted once it has been noncurrent for NoncurrentDays (counted from when the next version was written), keeping the newest NewerNoncurrentVersions noncurrent versions whatever their age.
AbortIncompleteMultipartUpload Uploads started more than DaysAfterInitiation days ago are aborted and their parts reclaimed.

Filters: <Prefix>, <Tag>, <ObjectSizeGreaterThan>, <ObjectSizeLessThan>, or several inside <And>; the legacy top-level <Prefix> is accepted. Tag and size filters don't apply to delete markers.

Guarantees. Lifecycle acts through the same delete path as a client, so a version under Object Lock retention or a legal hold is left alone (and counted in iolake_lifecycle_actions_total{action="locked"}), the listing stays right, and space is reclaimed. Every action is an audit event by principal lifecycle:<rule id>.

When. One gateway at a time scans (a lease held in meta); --lifecycle-interval-secs sets how often (default hourly). An object becomes eligible once its age passes the rule's days; it is acted on at the next scan.

Refused: Transition / NoncurrentVersionTransition (InvalidStorageClass — there is one storage class), Days ≤ 0, a Date not at midnight UTC, rules without an action, duplicate IDs, Filter and Prefix together.


CORS

PUT    /{bucket}?cors
GET    /{bucket}?cors      → 404 NoSuchCORSConfiguration when none
DELETE /{bucket}?cors
<CORSConfiguration>
  <CORSRule>
    <ID>web-app</ID>
    <AllowedOrigin>https://app.example.com</AllowedOrigin>
    <AllowedOrigin>https://*.example.com</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
    <ExposeHeader>ETag</ExposeHeader>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>
  • Preflights (OPTIONS on a bucket or object) are answered without credentials — browsers never sign them — from the configuration alone: the first rule allowing the origin, the method and every requested header. No match (or no configuration): 403 AccessForbidden. A preflight reveals nothing but whether the bucket exists.
  • Actual requests with an Origin that a rule allows get Access-Control-Allow-Origin (the origin itself, with Access-Control-Allow-Credentials: true, unless the rule's origin is exactly *), -Allow-Methods, -Expose-Headers, -Max-Age and Vary: Origin — error responses included, so a browser can read S3's errors. CORS only adds headers: every request is still authenticated and authorized.
  • Up to 100 rules; methods GET PUT POST DELETE HEAD; at most one * in an origin or header pattern.

POST Object (browser uploads)

A browser uploads straight into a bucket with an HTML form, signed by your backend — the S3 form-upload API, so boto3's generate_presigned_post and every S3 form library work.

post = s3.generate_presigned_post(
    "uploads", "user-42/${filename}",
    Conditions=[["starts-with", "$key", "user-42/"], ["content-length-range", 1, 10_000_000]],
    ExpiresIn=600)
# hand post["url"] and post["fields"] to the browser's form
  • POST /{bucket} with multipart/form-data; fields key (with ${filename}), policy, x-amz-algorithm (AWS4-HMAC-SHA256 only), x-amz-credential, x-amz-date, x-amz-signature, optional x-amz-security-token, Content-Type, Cache-Control, Content-Disposition, x-amz-meta-*, success_action_status (200, 201, 204), success_action_redirect, and file last.
  • The policy must not have expired, and every form field (except the signature, file, policy and x-ignore-*) must be covered by one of its conditions: exact match, eq, starts-with, content-length-range. The signature and policy are checked before any of the file is read, and the file is read only up to the policy's size limit (and 100 MiB: a form's file is held whole, unlike a PUT's body).
  • Authorization is the signer's, exactly as for a PUT: scoped keys, the tenant boundary, policies and Block Public Access all apply. The object is written by the same path a PUT takes (default encryption, Object Lock, versioning, checksums).
  • A form without a signature is anonymous: it needs a bucket policy allowing anonymous s3:PutObject, and public access not blocked.
  • Responses: 204 by default, 200, 201 with a PostResponse document, or 303 to success_action_redirect. Policy violations: 403 AccessDenied ("Invalid according to Policy: …"); a file outside content-length-range: 400 EntityTooLarge / EntityTooSmall.

Public Access Block

See Public Access for what the flags do and the tenant- and cluster-wide equivalents.

PUT /{bucket}?publicAccessBlock set the bucket's block (PublicAccessBlockConfiguration XML; absent flags are false)
GET /{bucket}?publicAccessBlock the bucket's own block; 404 NoSuchPublicAccessBlockConfiguration if none
DELETE /{bucket}?publicAccessBlock remove it (tenant and cluster blocks still apply)
GET /{bucket}?policyStatus <PolicyStatus><IsPublic> — whether the bucket policy makes it public

PutBucketPolicy answers 403 AccessDenied for a public policy while BlockPublicPolicy holds at any level.


ACLs

Buckets are bucket owner enforced. GetBucketAcl / GetObjectAcl answer the owner with FULL_CONTROL; GetBucketOwnershipControls answers BucketOwnerEnforced. An ACL write naming only that is accepted; any other is 400 AccessControlListNotSupported. See ACLs.


Versioning

PutBucketVersioning

Enable/suspend versioning.

PUT /{bucket}?versioning

Request Body:

<VersioningConfiguration>
    <Status>Enabled</Status>
</VersioningConfiguration>

GetBucketVersioning

Get versioning status.

GET /{bucket}?versioning

ListObjectVersions

List object versions.

GET /{bucket}?versions

Bucket settings on the admin API

A bucket's versioning, Object Lock, default encryption, lifecycle and replication can also be read and changed on the admin API, as JSON. The console uses these routes: it signs in with a session cookie, which the S3 path doesn't accept.

Route Methods S3 equivalent
/_admin/buckets/{name}/versioning GET, PUT ?versioning
/_admin/buckets/{name}/object-lock GET, PUT ?object-lock
/_admin/buckets/{name}/encryption GET, PUT, DELETE ?encryption
/_admin/buckets/{name}/lifecycle GET, PUT, DELETE ?lifecycle
/_admin/buckets/{name}/replication GET, PUT, DELETE ?replication

Who. The system admin, for any bucket; a tenant admin, for its own tenant's buckets only. A bucket outside any tenant is the system admin's alone. This is the rule every per-bucket admin route applies (/_admin/buckets/{name}/policy, /owner, /objects). The caller may sign with SigV4 or use a console session; a key scoped to a bucket or prefix is refused.

What. Each body is S3's configuration document as JSON. Element names are the ones S3 uses in XML, and a repeated element (Rule, Tag) is an array under that name. A setting goes through the same code as its S3 call, so S3's checks and limits apply as described above: lifecycle's refusals, Object Lock needing versioning, replication needing versioning and an existing target.

  • GET answers 200 with the document.
  • PUT takes the document and answers 204.
  • DELETE answers 204. S3 has no DELETE for versioning or Object Lock, and neither do these routes.

Errors are {"code": "...", "message": "..."}, with the status and code S3 gives. A setting the bucket doesn't have is 404 with S3's code for it. A bucket that doesn't exist is 404 NoSuchBucket. A body that isn't JSON for the setting is 400 MalformedJSON. A caller without credentials gets 401, and a caller who isn't an admin of the bucket's tenant gets 403 AccessDenied.

Setting Not configured
versioning never 404: a bucket never versioned answers {}, as S3 answers an empty VersioningConfiguration
object-lock 404 ObjectLockConfigurationNotFoundError
encryption 404 ServerSideEncryptionConfigurationNotFoundError
lifecycle 404 NoSuchLifecycleConfiguration
replication 404 ReplicationConfigurationNotFoundError

Examples.

# Versioning: {"Status": "Enabled"} or {"Status": "Suspended"}
curl -X PUT -b "iolake-session=$SESSION" -H 'Content-Type: application/json' \
  -d '{"Status": "Enabled"}' http://localhost:9000/_admin/buckets/photos/versioning
# 204
curl -b "iolake-session=$SESSION" http://localhost:9000/_admin/buckets/photos/versioning
# {"Status":"Enabled"}
// PUT /_admin/buckets/photos/object-lock (the bucket must be versioned)
{"ObjectLockEnabled": "Enabled",
 "Rule": {"DefaultRetention": {"Mode": "GOVERNANCE", "Days": 7}}}

// PUT /_admin/buckets/photos/encryption
{"Rule": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}

// PUT /_admin/buckets/photos/lifecycle
{"Rule": [{"ID": "expire-logs", "Filter": {"Prefix": "logs/"},
           "Status": "Enabled", "Expiration": {"Days": 30}}]}

// PUT /_admin/buckets/photos/replication
{"Rule": [{"ID": "dr", "Status": "Enabled", "Filter": {"Prefix": "docs/"},
           "DeleteMarkerReplication": {"Status": "Disabled"},
           "Destination": {"Bucket": "arn:obio:replication:::site-b"}}]}

A GET answers the document as stored, in the form S3's GET answers it. For example, a replication rule comes back with its filter inside And, and its Priority and DeleteMarkerReplication set.

GET /_admin/buckets gives each bucket's Object Lock as object_lock: {"enabled": false}, or {"enabled": true} with a default_retention (mode, and days or years) when it has one:

{"name": "photos", "object_lock": {"enabled": true,
  "default_retention": {"mode": "GOVERNANCE", "days": 7}}, "...": "..."}

S3 Select

SelectObjectContent: run SQL over one CSV, JSON or Parquet object and get back only what it selects.

POST /{bucket}/{key}?select&select-type=2
r = s3.select_object_content(
    Bucket="logs", Key="2026/10/app.csv.gz",
    ExpressionType="SQL",
    Expression="SELECT s.ts, s.msg FROM S3Object s WHERE s.level = 'ERROR' LIMIT 100",
    InputSerialization={"CSV": {"FileHeaderInfo": "USE"}, "CompressionType": "GZIP"},
    OutputSerialization={"JSON": {}})
for event in r["Payload"]:
    if "Records" in event:
        print(event["Records"]["Payload"].decode(), end="")
  • Input: CSV (FileHeaderInfo, delimiters, quote and escape, Comments, AllowQuotedRecordDelimiter), JSON (DOCUMENT or LINES), Parquet. CSV and JSON may be GZIP or BZIP2.
  • Output: CSV (delimiters, quoting ASNEEDED/ALWAYS) or JSON.
  • SQL: AWS's subset — SELECT list with aliases, FROM S3Object[*].path alias, WHERE, LIMIT, CAST, string, date and conditional functions, COUNT/SUM/AVG/MIN/MAX, with AWS's meaning. RGW's additions that AWS would refuse also work (double-quoted strings, INT(x), aliases in WHERE); where RGW gives a valid AWS statement another meaning, IOLake follows AWS: in LIKE, [ and ] are themselves (RGW: [a-z] sets), and FROM S3Object[*].arr gives the array as one record (arr[*] reads its elements; RGW reads them in both). See the design.
  • Response: AWS's event stream: Records, Progress when RequestProgress is on, Stats, End; a failure after the stream began is an error event.
  • ScanRange on uncompressed CSV, JSON Lines and Parquet.
  • Permission: s3:GetObject on the key. SSE-C objects need their key headers, as for a GET.
  • Limits: a record or result row at most 1 MB; the expression at most 256 KB. The object is read in ranges, never whole into memory.

Not Yet Implemented

A request naming one of these sub-resources is refused with 501 NotImplemented rather than handled as something else (GET /b?website used to list the bucket):

Sub-resource S3 calls
?website Get/Put/DeleteBucketWebsite
?notification Get/PutBucketNotificationConfiguration
?inventory, ?analytics, ?metrics, ?intelligent-tiering the bucket's inventory, analytics, metrics and Intelligent-Tiering configurations
?accelerate Get/PutBucketAccelerateConfiguration
?requestPayment Get/PutBucketRequestPayment
?session CreateSession (S3 Express directory buckets)
?metadataTable, ?metadataConfiguration S3 Metadata configurations
?abac Get/PutBucketAbac
?restore, ?torrent RestoreObject, GetObjectTorrent

A GET on a bucket is also 501 when it names, without a value, any other parameter that is neither a sub-resource the gateway answers nor a listing parameter (GET /b?made-up): S3 names sub-resources that way, and answering one with the bucket's listing would report a success that wasn't one. Parameters with a value, such as an SDK's x-id=ListObjects, are left to the listing, as S3 does. Bucket logging (?logging) is supported: see Bucket Logging.

  • SigV2 — refused with 400; use SigV4