S3 API Operations¶
Supported S3 API operations in IOLake.
Addressing and signing¶
- Path-style always; virtual-hosted where the gateway has an S3
domain. A request names its bucket in the path
(
https://s3.example.com/bucket/key), or, once the operator has set the gateway's S3 domain, in the host (https://bucket.s3.example.com/key): see Virtual-hosted addressing below. Path style works with any setup, so it is the safe choice for a client that doesn't know: boto3Config(s3={"addressing_style": "path"}), the AWS CLIaws configure set default.s3.addressing_style path, the Java and Go SDKsforcePathStyle/UsePathStyle, Iceberg and Sparks3.path-style-access=true. - SigV4 only, for requests and presigned URLs; SigV2 is refused with
400. Against a custom endpoint, boto3 withoutsignature_version="s3v4"and AWS CLI v1'saws s3 presignproduce SigV2 presigned URLs: setConfig(signature_version="s3v4")in boto3, andaws configure set default.s3.signature_version s3v4for the CLI. - Overload answers
503 SlowDown(withRetry-After), when a gateway is at its memory budget or its limit of requests in flight; SDKs retry it with backoff. A request body that sends nothing for 30 s gets400 RequestTimeout(gateway memory). - A write with no room answers
507 StorageFull(MinIO'sXMinioStorageFull; AWS has no such error): the OSDs it needs are past their full ratio, or the filesystem under their disks or metadata is full (ENOSPC, EDQUOT). SDKs don't retry it, as they do a503(full disks).
Virtual-hosted addressing¶
The gateway reads a bucket from the host name once it is told the
domains buckets are named under: --s3-domain on the gateway (repeat
it, or separate domains with commas; env IOLAKE_S3_DOMAIN),
--s3-domain on iolake-aio and iolake-install, and
gateway.s3Domains in the Helm chart
(configuration). With
--s3-domain s3.example.com, a request's Host is read as S3 reads it:
Host |
Addressing | Bucket |
|---|---|---|
photos.s3.example.com |
virtual-hosted; the path is the key | photos |
my.photos.s3.example.com |
virtual-hosted; a bucket name may hold dots | my.photos |
s3.example.com |
path-style | the path's first segment |
an address, localhost, any name not under a domain |
path-style | the path's first segment |
The port is ignored and case doesn't matter. With several domains, the
longest one a host is under is used (s3.example.com before
example.com). Everything after works on the bucket from the host:
signatures (the path and host signed are the ones sent), presigned URLs,
bucket policies and IAM policies, CORS, metrics, the audit log. Bucket
operations go to the bucket's own host: PUT / creates the bucket,
GET /?list-type=2 lists it, POST /?delete is DeleteObjects,
DELETE / deletes it. The domain itself stays path-style, so the admin
API, the console, the Iceberg catalog and Delta Sharing are reached on it
as before.
Two things the gateway doesn't provide, for each domain:
- Wildcard DNS:
*.s3.example.comresolving to the gateway (its load balancer or ingress), and, behind an ingress, a host rule for the wildcard (gateway.ingress.hosts,host: "*.s3.example.com"). - A TLS certificate covering
*.s3.example.comwhere TLS ends. A wildcard covers one label only, so a bucket name with dots (my.photos.s3.example.com) fails certificate checks over HTTPS, as on AWS: address such buckets path-style.
The domains are not taken from --external-endpoint: that would make
every name under the endpoint's host a bucket, and a gateway reached at a
name under it (gw1.s3.example.com) would stop working.
What's refused rather than guessed. A request whose bucket can't be told for certain gets an S3 error, never another request's answer:
400 InvalidBucketName: a host under a domain whose bucket part isn't a valid bucket name (Bad_Name.s3.example.com).400 InvalidRequest: on a path-style host, a bucket operation on/—PUT /,DELETE /,HEAD /,POST /?delete, a form upload to/, orGET /with a listing's parameters (list-type,encoding-type,location, …). Only a client that put the bucket in the host sends these; read path-style they would name no bucket, and a listing would have come back as the caller's buckets. The message says to set--s3-domain(or names the domains the host isn't under), or to use path style. ListBuckets (GET /, withmax-buckets,prefix,continuation-token,bucket-region), the IAM and STS query API (Action=) and CORS preflights on/are unchanged.
Without a domain the gateway can't tell bucket.s3.example.com from
s3.example.com, so a request that is also valid path-style is read
path-style: a bare GET / is ListBuckets, and /key is bucket key.
If any client addresses buckets by host name, set --s3-domain.
Bucket Operations¶
ListBuckets¶
List all buckets owned by the authenticated user.
GET /
Response:
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult>
<Owner>
<ID>user-id</ID>
<DisplayName>User Name</DisplayName>
</Owner>
<Buckets>
<Bucket>
<Name>my-bucket</Name>
<CreationDate>2024-01-15T10:30:00.000Z</CreationDate>
</Bucket>
</Buckets>
</ListAllMyBucketsResult>
CreateBucket¶
Create a new bucket.
PUT /{bucket}
Headers:
- x-amz-acl: only private or bucket-owner-full-control (ACLs are
owner-enforced)
- x-amz-bucket-object-lock-enabled: true: a lock bucket (versioned)
- x-iolake-pool: <pool>: the pool its data goes to (default: the
tenant's default pool, else the cluster's default placement). A tenant
may use only its default and allowed pools; the pool is fixed for the
bucket's life
Response: 200 OK
A new bucket starts with public access blocked, and with no configuration from any earlier bucket of the same name.
DeleteBucket¶
Delete an empty bucket.
DELETE /{bucket}
Response: 204 No Content
Errors:
- BucketNotEmpty: the bucket holds objects, or (versioned) noncurrent
versions or delete markers
Deleting a bucket also removes its policy, object lock, lifecycle, encryption and public access settings, in the same commit.
HeadBucket¶
Check if a bucket exists and you have access.
HEAD /{bucket}
Response: 200 OK (bucket exists) or 404 Not Found
GetBucketLocation¶
Get the region of a bucket: the gateway's --region
(configuration), which every bucket
is in. Authorized as s3:GetBucketLocation.
GET /{bucket}?location
Response (a gateway started with --region eu-west-1):
<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">eu-west-1</LocationConstraint>
In us-east-1, the default, the constraint is empty, as S3 answers it;
SDKs read an empty constraint as us-east-1:
<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>
A bucket that doesn't exist is 404 NoSuchBucket.
Object Operations¶
PutObject¶
Upload an object.
PUT /{bucket}/{key}
Headers:
- Content-Type: MIME type
- Content-Length: Object size
- x-amz-meta-*: User metadata
- Cache-Control, Content-Disposition, Content-Encoding,
Content-Language, Expires: kept with the object and served back
(and kept by a copy)
- If-None-Match: *: write only if no object has the key (412
PreconditionFailed if one does) — a create that can't overwrite
- If-Match: <etag>: write only if the current object has this ETag
(412, or 404 NoSuchKey if there's none) — compare-and-swap
Conditional writes are checked and applied atomically: of two writers
racing with If-None-Match: *, exactly one wins. CompleteMultipartUpload
takes the same headers.
A body may be up to 5 GiB, as S3 allows (large uploads);
a larger one gets 400 EntityTooLarge before any of it is read.
Response:
HTTP/1.1 200 OK
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Example:
curl -X PUT \
-H "Content-Type: text/plain" \
-d "Hello World" \
http://localhost:9000/my-bucket/hello.txt
GetObject¶
Download an object.
GET /{bucket}/{key}
Headers:
- Range: Byte range (optional)
Response Headers:
- Content-Type: Object MIME type
- Content-Length: Object size
- ETag: Object hash
- Last-Modified: Last modification time
Range Request:
curl -H "Range: bytes=0-99" http://localhost:9000/my-bucket/file.txt
The object is streamed stripe by stripe, never held whole in the
gateway. A failure reading the first stripe is an error status; one after
the headers have gone out closes the connection short of
Content-Length (never with wrong bytes), which clients see as a short
read and retry.
HeadObject¶
Get object metadata without body.
HEAD /{bucket}/{key}
Response Headers:
- Content-Type: Object MIME type
- Content-Length: Object size
- ETag: Object hash
- x-amz-meta-*: User metadata
DeleteObject¶
Delete an object.
DELETE /{bucket}/{key}
Response: 204 No Content
CopyObject¶
Copy an object.
PUT /{bucket}/{key}
x-amz-copy-source: /source-bucket/source-key
Response:
<?xml version="1.0" encoding="UTF-8"?>
<CopyObjectResult>
<LastModified>2024-01-15T10:30:00.000Z</LastModified>
<ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
</CopyObjectResult>
A copy whose source and destination are encrypted alike shares the
source's stripes, at any size. A copy that changes the encryption reads
the source and writes it again, streamed as a large PUT is, so its source
(or, for UploadPartCopy, its range) may be at most 5 GiB, as S3 allows; a
larger one gets 400 InvalidRequest, as S3 answers.
ListObjectsV2¶
List objects in a bucket.
GET /{bucket}?list-type=2
Parameters:
- prefix: Filter by prefix
- delimiter: Group by delimiter
- max-keys: Max results (default: 1000)
- continuation-token: Pagination token
- start-after: Start listing after this key
Response:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult>
<Name>my-bucket</Name>
<Prefix></Prefix>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>file1.txt</Key>
<LastModified>2024-01-15T10:30:00.000Z</LastModified>
<ETag>"abc123"</ETag>
<Size>1024</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
</ListBucketResult>
Multipart Upload¶
Multipart upload allows uploading large objects in parts. It is required for objects over 5 GiB, the most a single PUT takes, as in S3.
Limits and Behavior¶
| Parameter | Value | Notes |
|---|---|---|
| Minimum part size | 5 MiB | Except the last part (400 EntityTooSmall at completion) |
| Maximum part size | 5 GiB | As S3; a larger part gets 400 EntityTooLarge before its body is read |
| Maximum single PUT | 5 GiB | As S3; a larger one gets 400 EntityTooLarge before its body is read |
| Maximum parts | 10,000 | |
| Maximum object size | 5 TiB | As S3; a completion whose parts make more gets 400 EntityTooLarge |
| Default AWS CLI part size | 8 MB | multipart_chunksize in the CLI's S3 config; up to 5 GiB |
Large uploads¶
A single PUT or an UploadPart over 16 MiB is not held in the gateway's
memory: its body is streamed to the OSDs stripe by stripe as it arrives,
each stripe erasure-coded (and encrypted, with SSE) and written while the
next is read, a couple of stripes in flight at once. However large the
upload, the gateway holds a few stripes of it (about 48 MiB with 4+2),
reserved from its memory budget
before the body is read: with no room, the request gets 503 SlowDown
and the client retries.
The upload is acknowledged (200) only once every stripe is on stable
storage at write quorum and the object's metadata is committed. The object
becomes visible only then, after the whole body has arrived and passed
every check it carries:
- its signed payload hash (
x-amz-content-sha256), computed over the bytes as they stream (400 XAmzContentSHA256Mismatch); - with
aws-chunked, each chunk's signature, checked as each chunk ends (403 SignatureDoesNotMatch), and the checksum trailer; Content-MD5(400 BadDigest) and anx-amz-checksum-*header or trailer (400 BadDigest).
A body that fails a check, or a client that goes away mid-body, stores
nothing: the shards already written are freed at once (as for any failed
write, a shard an OSD can't be reached to free is retried for a while). A
gateway that stops mid-upload leaves its shards allocated until the
orphan scan on the roadmap reclaims them. An upload that meets a
placement group whose acting set changed under it gets 503, which
clients retry; a body small enough to be held whole is placed again
without the client seeing it.
Multi-Stripe Chunking¶
IOLake uses a 4MB internal block size. When a part exceeds 4MB, it is automatically split into multiple stripes:
8 MB Part Upload:
┌─────────────────────────────────────────────────────┐
│ 8 MB Part Data │
└─────────────────────────────────────────────────────┘
│
Split into ~4MB stripes
│
┌───────────────┴───────────────┐
▼ ▼
┌─────────┐ ┌─────────┐
│ Stripe 0│ (4 MB) │ Stripe 1│ (4 MB)
└────┬────┘ └────┬────┘
│ │
EC Encode (4+2) EC Encode (4+2)
│ │
6 shards 6 shards
Each stripe is independently erasure-coded and distributed across storage nodes.
CreateMultipartUpload¶
Initiate a multipart upload.
POST /{bucket}/{key}?uploads
Response:
<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult>
<Bucket>my-bucket</Bucket>
<Key>large-file.zip</Key>
<UploadId>abc123</UploadId>
</InitiateMultipartUploadResult>
UploadPart¶
Upload a part. Parts can be uploaded in any order and in parallel.
PUT /{bucket}/{key}?partNumber={n}&uploadId={id}
Parameters:
partNumber: 1 to 10,000uploadId: From CreateMultipartUpload response
Response:
HTTP/1.1 200 OK
ETag: "part-etag"
Example with curl:
# Upload part 1
curl -X PUT \
"http://localhost:9000/my-bucket/large-file?partNumber=1&uploadId=abc123" \
--data-binary @part1.bin
CompleteMultipartUpload¶
Complete the upload by providing the list of parts in order.
POST /{bucket}/{key}?uploadId={id}
Request Body:
<CompleteMultipartUpload>
<Part>
<PartNumber>1</PartNumber>
<ETag>"part1-etag"</ETag>
</Part>
<Part>
<PartNumber>2</PartNumber>
<ETag>"part2-etag"</ETag>
</Part>
</CompleteMultipartUpload>
Response:
<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult>
<Location>http://localhost:9000/my-bucket/large-file</Location>
<Bucket>my-bucket</Bucket>
<Key>large-file</Key>
<ETag>"abc123-2"</ETag>
</CompleteMultipartUploadResult>
The final ETag format is "<md5-of-part-md5s>-<part-count>".
AbortMultipartUpload¶
Abort an in-progress upload and delete uploaded parts.
DELETE /{bucket}/{key}?uploadId={id}
Response: 204 No Content
ListParts¶
List uploaded parts for an in-progress multipart upload.
GET /{bucket}/{key}?uploadId={id}
Parameters:
max-parts: Maximum parts to return (default: 1000)part-number-marker: Start listing after this part number
Response:
<?xml version="1.0" encoding="UTF-8"?>
<ListPartsResult>
<Bucket>my-bucket</Bucket>
<Key>large-file</Key>
<UploadId>abc123</UploadId>
<MaxParts>1000</MaxParts>
<IsTruncated>false</IsTruncated>
<Part>
<PartNumber>1</PartNumber>
<LastModified>2024-01-15T10:30:00.000Z</LastModified>
<ETag>"part1-etag"</ETag>
<Size>8388608</Size>
</Part>
</ListPartsResult>
ListMultipartUploads¶
List in-progress multipart uploads for a bucket.
GET /{bucket}?uploads
Parameters:
prefix: Filter by key prefixmax-uploads: Maximum uploads to return (default: 1000)
Response:
<?xml version="1.0" encoding="UTF-8"?>
<ListMultipartUploadsResult>
<Bucket>my-bucket</Bucket>
<MaxUploads>1000</MaxUploads>
<IsTruncated>false</IsTruncated>
<Upload>
<Key>large-file</Key>
<UploadId>abc123</UploadId>
<Initiated>2024-01-15T10:30:00.000Z</Initiated>
</Upload>
</ListMultipartUploadsResult>
Example: Upload Large File with AWS CLI¶
# AWS CLI automatically uses multipart upload for files > 8MB
aws --endpoint-url http://localhost:9000 s3 cp large-file.zip s3://my-bucket/
# Configure part size (minimum 5MB)
aws configure set default.s3.multipart_chunksize 16MB
aws --endpoint-url http://localhost:9000 s3 cp large-file.zip s3://my-bucket/
# Manual multipart upload
aws --endpoint-url http://localhost:9000 s3api create-multipart-upload \
--bucket my-bucket --key large-file.zip
# Upload parts
aws --endpoint-url http://localhost:9000 s3api upload-part \
--bucket my-bucket --key large-file.zip \
--part-number 1 --upload-id $UPLOAD_ID --body part1.bin
# Complete upload
aws --endpoint-url http://localhost:9000 s3api complete-multipart-upload \
--bucket my-bucket --key large-file.zip \
--upload-id $UPLOAD_ID \
--multipart-upload file://parts.json
Bucket Policy¶
PutBucketPolicy¶
Set bucket policy.
PUT /{bucket}?policy
Request Body:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-bucket/*"
}
]
}
GetBucketPolicy¶
Get bucket policy.
GET /{bucket}?policy
DeleteBucketPolicy¶
Delete bucket policy.
DELETE /{bucket}?policy
Lifecycle¶
PUT /{bucket}?lifecycle
GET /{bucket}?lifecycle → 404 NoSuchLifecycleConfiguration when none
DELETE /{bucket}?lifecycle
<LifecycleConfiguration>
<Rule>
<ID>logs</ID>
<Filter><Prefix>logs/</Prefix></Filter>
<Status>Enabled</Status>
<Expiration><Days>30</Days></Expiration>
</Rule>
<Rule>
<ID>old-versions</ID>
<Filter/>
<Status>Enabled</Status>
<NoncurrentVersionExpiration>
<NoncurrentDays>7</NoncurrentDays>
<NewerNoncurrentVersions>3</NewerNoncurrentVersions>
</NoncurrentVersionExpiration>
<AbortIncompleteMultipartUpload><DaysAfterInitiation>2</DaysAfterInitiation></AbortIncompleteMultipartUpload>
</Rule>
</LifecycleConfiguration>
| action | what happens |
|---|---|
Expiration Days / Date |
The current version expires. Unversioned: deleted. Versioned or suspended: a delete marker is added, as a DELETE without a version id would — the data stays as a noncurrent version. |
Expiration ExpiredObjectDeleteMarker |
A delete marker that is the key's only version is removed. Days does this too in a versioned bucket. A marker with versions behind it is never removed: that would bring the object back. |
NoncurrentVersionExpiration |
A noncurrent version is deleted once it has been noncurrent for NoncurrentDays (counted from when the next version was written), keeping the newest NewerNoncurrentVersions noncurrent versions whatever their age. |
AbortIncompleteMultipartUpload |
Uploads started more than DaysAfterInitiation days ago are aborted and their parts reclaimed. |
Filters: <Prefix>, <Tag>, <ObjectSizeGreaterThan>, <ObjectSizeLessThan>, or several inside <And>; the legacy top-level <Prefix> is accepted. Tag and size filters don't apply to delete markers.
Guarantees. Lifecycle acts through the same delete path as a client, so a version under Object Lock retention or a legal hold is left alone (and counted in iolake_lifecycle_actions_total{action="locked"}), the listing stays right, and space is reclaimed. Every action is an audit event by principal lifecycle:<rule id>.
When. One gateway at a time scans (a lease held in meta); --lifecycle-interval-secs sets how often (default hourly). An object becomes eligible once its age passes the rule's days; it is acted on at the next scan.
Refused: Transition / NoncurrentVersionTransition (InvalidStorageClass — there is one storage class), Days ≤ 0, a Date not at midnight UTC, rules without an action, duplicate IDs, Filter and Prefix together.
CORS¶
PUT /{bucket}?cors
GET /{bucket}?cors → 404 NoSuchCORSConfiguration when none
DELETE /{bucket}?cors
<CORSConfiguration>
<CORSRule>
<ID>web-app</ID>
<AllowedOrigin>https://app.example.com</AllowedOrigin>
<AllowedOrigin>https://*.example.com</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>ETag</ExposeHeader>
<MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>
</CORSConfiguration>
- Preflights (
OPTIONSon a bucket or object) are answered without credentials — browsers never sign them — from the configuration alone: the first rule allowing the origin, the method and every requested header. No match (or no configuration):403 AccessForbidden. A preflight reveals nothing but whether the bucket exists. - Actual requests with an
Originthat a rule allows getAccess-Control-Allow-Origin(the origin itself, withAccess-Control-Allow-Credentials: true, unless the rule's origin is exactly*),-Allow-Methods,-Expose-Headers,-Max-AgeandVary: Origin— error responses included, so a browser can read S3's errors. CORS only adds headers: every request is still authenticated and authorized. - Up to 100 rules; methods
GET PUT POST DELETE HEAD; at most one*in an origin or header pattern.
POST Object (browser uploads)¶
A browser uploads straight into a bucket with an HTML form, signed by your
backend — the S3 form-upload API, so boto3's generate_presigned_post
and every S3 form library work.
post = s3.generate_presigned_post(
"uploads", "user-42/${filename}",
Conditions=[["starts-with", "$key", "user-42/"], ["content-length-range", 1, 10_000_000]],
ExpiresIn=600)
# hand post["url"] and post["fields"] to the browser's form
POST /{bucket}withmultipart/form-data; fieldskey(with${filename}),policy,x-amz-algorithm(AWS4-HMAC-SHA256only),x-amz-credential,x-amz-date,x-amz-signature, optionalx-amz-security-token,Content-Type,Cache-Control,Content-Disposition,x-amz-meta-*,success_action_status(200, 201, 204),success_action_redirect, andfilelast.- The policy must not have expired, and every form field (except the
signature,
file,policyandx-ignore-*) must be covered by one of its conditions: exact match,eq,starts-with,content-length-range. The signature and policy are checked before any of the file is read, and the file is read only up to the policy's size limit (and 100 MiB: a form's file is held whole, unlike a PUT's body). - Authorization is the signer's, exactly as for a PUT: scoped keys, the tenant boundary, policies and Block Public Access all apply. The object is written by the same path a PUT takes (default encryption, Object Lock, versioning, checksums).
- A form without a signature is anonymous: it needs a bucket policy
allowing anonymous
s3:PutObject, and public access not blocked. - Responses: 204 by default, 200, 201 with a
PostResponsedocument, or 303 tosuccess_action_redirect. Policy violations:403 AccessDenied("Invalid according to Policy: …"); a file outsidecontent-length-range:400 EntityTooLarge/EntityTooSmall.
Public Access Block¶
See Public Access for what the flags do and the tenant- and cluster-wide equivalents.
PUT /{bucket}?publicAccessBlock |
set the bucket's block (PublicAccessBlockConfiguration XML; absent flags are false) |
GET /{bucket}?publicAccessBlock |
the bucket's own block; 404 NoSuchPublicAccessBlockConfiguration if none |
DELETE /{bucket}?publicAccessBlock |
remove it (tenant and cluster blocks still apply) |
GET /{bucket}?policyStatus |
<PolicyStatus><IsPublic> — whether the bucket policy makes it public |
PutBucketPolicy answers 403 AccessDenied for a public policy while
BlockPublicPolicy holds at any level.
ACLs¶
Buckets are bucket owner enforced. GetBucketAcl / GetObjectAcl answer
the owner with FULL_CONTROL; GetBucketOwnershipControls answers
BucketOwnerEnforced. An ACL write naming only that is accepted; any other
is 400 AccessControlListNotSupported. See ACLs.
Versioning¶
PutBucketVersioning¶
Enable/suspend versioning.
PUT /{bucket}?versioning
Request Body:
<VersioningConfiguration>
<Status>Enabled</Status>
</VersioningConfiguration>
GetBucketVersioning¶
Get versioning status.
GET /{bucket}?versioning
ListObjectVersions¶
List object versions.
GET /{bucket}?versions
Bucket settings on the admin API¶
A bucket's versioning, Object Lock, default encryption, lifecycle and replication can also be read and changed on the admin API, as JSON. The console uses these routes: it signs in with a session cookie, which the S3 path doesn't accept.
| Route | Methods | S3 equivalent |
|---|---|---|
/_admin/buckets/{name}/versioning |
GET, PUT |
?versioning |
/_admin/buckets/{name}/object-lock |
GET, PUT |
?object-lock |
/_admin/buckets/{name}/encryption |
GET, PUT, DELETE |
?encryption |
/_admin/buckets/{name}/lifecycle |
GET, PUT, DELETE |
?lifecycle |
/_admin/buckets/{name}/replication |
GET, PUT, DELETE |
?replication |
Who. The system admin, for any bucket; a tenant admin, for its own
tenant's buckets only. A bucket outside any tenant is the system admin's
alone. This is the rule every per-bucket admin route applies
(/_admin/buckets/{name}/policy, /owner, /objects). The caller may sign
with SigV4 or use a console session; a key scoped to a bucket or prefix
is refused.
What. Each body is S3's configuration document as JSON. Element names
are the ones S3 uses in XML, and a repeated element (Rule, Tag) is an
array under that name. A setting goes through the same code as its S3 call,
so S3's checks and limits apply as described above: lifecycle's refusals,
Object Lock needing versioning, replication needing versioning and an
existing target.
GETanswers200with the document.PUTtakes the document and answers204.DELETEanswers204. S3 has no DELETE for versioning or Object Lock, and neither do these routes.
Errors are {"code": "...", "message": "..."}, with the status and
code S3 gives. A setting the bucket doesn't have is 404 with S3's code
for it. A bucket that doesn't exist is 404 NoSuchBucket. A body that
isn't JSON for the setting is 400 MalformedJSON. A caller without
credentials gets 401, and a caller who isn't an admin of the bucket's
tenant gets 403 AccessDenied.
| Setting | Not configured |
|---|---|
| versioning | never 404: a bucket never versioned answers {}, as S3 answers an empty VersioningConfiguration |
| object-lock | 404 ObjectLockConfigurationNotFoundError |
| encryption | 404 ServerSideEncryptionConfigurationNotFoundError |
| lifecycle | 404 NoSuchLifecycleConfiguration |
| replication | 404 ReplicationConfigurationNotFoundError |
Examples.
# Versioning: {"Status": "Enabled"} or {"Status": "Suspended"}
curl -X PUT -b "iolake-session=$SESSION" -H 'Content-Type: application/json' \
-d '{"Status": "Enabled"}' http://localhost:9000/_admin/buckets/photos/versioning
# 204
curl -b "iolake-session=$SESSION" http://localhost:9000/_admin/buckets/photos/versioning
# {"Status":"Enabled"}
// PUT /_admin/buckets/photos/object-lock (the bucket must be versioned)
{"ObjectLockEnabled": "Enabled",
"Rule": {"DefaultRetention": {"Mode": "GOVERNANCE", "Days": 7}}}
// PUT /_admin/buckets/photos/encryption
{"Rule": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}
// PUT /_admin/buckets/photos/lifecycle
{"Rule": [{"ID": "expire-logs", "Filter": {"Prefix": "logs/"},
"Status": "Enabled", "Expiration": {"Days": 30}}]}
// PUT /_admin/buckets/photos/replication
{"Rule": [{"ID": "dr", "Status": "Enabled", "Filter": {"Prefix": "docs/"},
"DeleteMarkerReplication": {"Status": "Disabled"},
"Destination": {"Bucket": "arn:obio:replication:::site-b"}}]}
A GET answers the document as stored, in the form S3's GET answers it.
For example, a replication rule comes back with its filter inside And,
and its Priority and DeleteMarkerReplication set.
GET /_admin/buckets gives each bucket's Object Lock as object_lock:
{"enabled": false}, or {"enabled": true} with a default_retention
(mode, and days or years) when it has one:
{"name": "photos", "object_lock": {"enabled": true,
"default_retention": {"mode": "GOVERNANCE", "days": 7}}, "...": "..."}
S3 Select¶
SelectObjectContent: run SQL over one CSV, JSON or Parquet object and get
back only what it selects.
POST /{bucket}/{key}?select&select-type=2
r = s3.select_object_content(
Bucket="logs", Key="2026/10/app.csv.gz",
ExpressionType="SQL",
Expression="SELECT s.ts, s.msg FROM S3Object s WHERE s.level = 'ERROR' LIMIT 100",
InputSerialization={"CSV": {"FileHeaderInfo": "USE"}, "CompressionType": "GZIP"},
OutputSerialization={"JSON": {}})
for event in r["Payload"]:
if "Records" in event:
print(event["Records"]["Payload"].decode(), end="")
- Input: CSV (
FileHeaderInfo, delimiters, quote and escape,Comments,AllowQuotedRecordDelimiter), JSON (DOCUMENTorLINES), Parquet. CSV and JSON may beGZIPorBZIP2. - Output: CSV (delimiters, quoting
ASNEEDED/ALWAYS) or JSON. - SQL: AWS's subset —
SELECTlist with aliases,FROM S3Object[*].path alias,WHERE,LIMIT,CAST, string, date and conditional functions,COUNT/SUM/AVG/MIN/MAX, with AWS's meaning. RGW's additions that AWS would refuse also work (double-quoted strings,INT(x), aliases inWHERE); where RGW gives a valid AWS statement another meaning, IOLake follows AWS: inLIKE,[and]are themselves (RGW:[a-z]sets), andFROM S3Object[*].arrgives the array as one record (arr[*]reads its elements; RGW reads them in both). See the design. - Response: AWS's event stream:
Records,ProgresswhenRequestProgressis on,Stats,End; a failure after the stream began is an error event. ScanRangeon uncompressed CSV, JSON Lines and Parquet.- Permission:
s3:GetObjecton the key. SSE-C objects need their key headers, as for a GET. - Limits: a record or result row at most 1 MB; the expression at most 256 KB. The object is read in ranges, never whole into memory.
Not Yet Implemented¶
A request naming one of these sub-resources is refused with 501
NotImplemented rather than handled as something else (GET /b?website
used to list the bucket):
| Sub-resource | S3 calls |
|---|---|
?website |
Get/Put/DeleteBucketWebsite |
?notification |
Get/PutBucketNotificationConfiguration |
?inventory, ?analytics, ?metrics, ?intelligent-tiering |
the bucket's inventory, analytics, metrics and Intelligent-Tiering configurations |
?accelerate |
Get/PutBucketAccelerateConfiguration |
?requestPayment |
Get/PutBucketRequestPayment |
?session |
CreateSession (S3 Express directory buckets) |
?metadataTable, ?metadataConfiguration |
S3 Metadata configurations |
?abac |
Get/PutBucketAbac |
?restore, ?torrent |
RestoreObject, GetObjectTorrent |
A GET on a bucket is also 501 when it names, without a value, any
other parameter that is neither a sub-resource the gateway answers nor a
listing parameter (GET /b?made-up): S3 names sub-resources that way, and
answering one with the bucket's listing would report a success that
wasn't one. Parameters with a value, such as an SDK's x-id=ListObjects,
are left to the listing, as S3 does. Bucket logging (?logging) is
supported: see Bucket Logging.
- SigV2 — refused with
400; use SigV4