Bucket Logging¶

S3 server access logging: every request on a bucket, as a line of AWS's access-log format, delivered as objects into a target bucket. Use it where a tool expects S3 access logs (Athena, log shippers). For a complete, real-time record of every request in the cluster, use the audit stream: bucket logging is per bucket, batched, and only for its owner's eyes.

Turning it on¶

Create the target bucket in the source bucket's tenant (or, for the operator, a bucket in the system tenant).
Let the logging service write there, with a bucket policy on the target:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "S3ServerAccessLogsPolicy",
    "Effect": "Allow",
    "Principal": {"Service": "logging.s3.amazonaws.com"},
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::access-logs/web/*",
    "Condition": {
      "ArnLike": {"aws:SourceArn": "arn:aws:s3:::web-assets"},
      "StringEquals": {"aws:SourceAccount": "<owner user id of web-assets>"}
    }
  }]
}

aws:SourceAccount is the source bucket's owner, its user id (what GET ?acl shows as the owner's ID). Wildcards work as in S3 (arn:aws:s3:::*, "*"). A grant to everyone ("Principal": "*") is not consent: the service must be named. 3. As the source bucket's owner:

aws s3api put-bucket-logging --bucket web-assets --bucket-logging-status '{
  "LoggingEnabled": {"TargetBucket": "access-logs", "TargetPrefix": "web/"}}'
aws s3api get-bucket-logging --bucket web-assets

For date-partitioned keys add "TargetObjectKeyFormat": {"PartitionedPrefix": {"PartitionDateSource": "EventTime"}} (or DeliveryTime).

Turn it off with an empty status: aws s3api put-bucket-logging --bucket web-assets --bucket-logging-status '{}'.

Refusal	Why
`403 AccessDenied` (only the bucket owner…)	Only the bucket's owner (or the system admin) configures logging, whatever policies grant
`403 AccessDenied` (Logging bucket … policy …)	The target's policy doesn't let the logging service put objects under the prefix for this bucket and owner
`400 InvalidTargetBucketForLogging`	The target doesn't exist, or is in another tenant
`400 InvalidArgument`	The target is the source, logs itself, or the prefix is over 512 bytes
`400 MalformedXML`	No `TargetBucket`, or a `PartitionDateSource` other than `EventTime`/`DeliveryTime`
`400 InvalidRequest` (no spool)	This gateway runs without a spool (below)

What arrives¶

One object per batch, named in UTC:

web/2026-10-05-07-08-09-3F2A9C1B7D4E6F80 (simple prefix), or
web/<owner>/<region>/web-assets/2026/10/05/2026-10-05-07-08-09-3F2A9C1B7D4E6F80 (partitioned).

Each line is AWS's format (LogFormat):

<owner> web-assets [05/Oct/2026:07:08:01 +0000] 10.0.0.7 arn:objectio:iam::acme:user/etl 6F1C0D3A9B2E4F7C8D1A2B3C4D5E6F70 REST.GET.OBJECT img/logo.png "GET /web-assets/img/logo.png HTTP/1.1" 200 - 5120 5120 12 9 "-" "aws-cli/2.17" - - SigV4 - AuthHeader s3.example.com - - -

The request ID is the x-amz-request-id the client got, and the id of the same request in the audit stream. Operations are named as S3 names them: REST.PUT.OBJECT, REST.GET.BUCKET, REST.COPY.OBJECT (and REST.COPY.OBJECT_GET in the source bucket's log), REST.POST.UPLOADS, REST.PUT.PART, REST.POST.UPLOAD, REST.POST.MULTI_OBJECT_DELETE plus a BATCH.DELETE.OBJECT line per key deleted. Refused requests are logged too, with their status and error code. Host id, cipher suite, TLS version, access point and "ACL required" are -.

Delivery¶

Records are kept on the gateway's disk before delivery, so a gateway that is killed or restarted loses none, and a request never waits for its log. A log object is written once its first record is --bucket-log-roll-secs old (default 300 s), or it holds --bucket-log-max-records (50 000) records or 16 MiB, and on shutdown. Delivery is at least once: after a crash a batch may be written twice (under two names); dedupe on the request id.

Before each object is written the target's consent is checked again: if the target is gone, in another tenant, or its policy no longer allows the logging service, those records are dropped (and counted). A target that fails otherwise (a full cluster, meta unavailable) is retried with backoff; records wait in the spool meanwhile.

A configuration change reaches other gateways within 10 seconds.

Settings¶

Gateway flag (env)	Default
`--bucket-log-spool` (`IOLAKE_BUCKET_LOG_SPOOL`)	`<--audit-spool>/bucket-logging`	Where records wait. A local directory that survives restarts. aio: `<data>/bucket-log-spool`
`--bucket-log-spool-max-bytes` (`IOLAKE_BUCKET_LOG_SPOOL_MAX_BYTES`)	1 GiB (Helm: 512 MiB, `gateway.audit.spool.bucketLogMaxBytes`)	Past it, new records are dropped and counted
`--bucket-log-roll-secs` (`IOLAKE_BUCKET_LOG_ROLL_SECS`)	300	How long a log object collects records
`--bucket-log-max-records` (`IOLAKE_BUCKET_LOG_MAX_RECORDS`)	50 000	A log object is written once it holds this many

With Helm the spool is in the audit spool's volume (gateway.audit.spool.enabled, on by default). A gateway with neither --bucket-log-spool nor --audit-spool refuses PutBucketLogging and drops the records of buckets logged through other gateways.

Metrics¶

Metric
`iolake_bucket_logging_records_total`	Records delivered
`iolake_bucket_logging_dropped_total{reason}`	Not delivered: `refused` (the target no longer consents), `target-refused` (a 4xx writing it), `spool-full`, `no-spool`, `lookup-failed` (a bucket's configuration couldn't be read), `unreadable`
`iolake_bucket_logging_delivery_failures_total`	Writes retried

Alert on dropped_total{reason=~"spool-full|no-spool|lookup-failed"} rising: those are records lost to the cluster, not refused by a target.

Differences from S3 and RGW¶

A target must be in the source's tenant (or the system's): no cross-tenant (cross-account) logging.
RGW's extensions are not supported: LoggingType (Journal), ObjectRollTime, RecordsBatchSize, POST ?logging, key filters, tenant:bucket targets.
What the gateway does on its own (lifecycle expiry, replication) is not logged.
The region in partitioned keys is the gateway's --region.