Audit Stream¶

Every request to the gateway becomes one audit event: who made it, from where, what it did, and how it ended. The operator streams all of them to their own systems (a SIEM, a log pipeline), and each tenant can stream its own to its own. S3 server access logging (?logging, per bucket, into a bucket) is supported too: see Bucket Logging. It is fed by the same layer, and its records carry the same request ids.

The event¶

One JSON object per request:

{
  "version": 1,
  "id": "6F1C0D3A9B2E4F7C8D1A2B3C4D5E6F70",
  "time": "2026-10-02T13:04:11.482Z",
  "api": "s3",
  "method": "PUT",
  "path": "/acme-data/reports/q3.parquet",
  "action": "s3:PutObject",
  "bucket": "acme-data",
  "key": "reports/q3.parquet",
  "bucket_tenant": "acme",
  "principal": {"user_id": "…", "arn": "arn:objectio:iam::acme:user/etl",
                "access_key": "AKIA…", "tenant": "acme", "auth": "Permanent"},
  "source": {"ip": "10.244.3.17", "endpoint": "internal", "user_agent": "aws-sdk-go-v2/1.30"},
  "status": 200,
  "request_bytes": 1048576,
  "response_bytes": 0,
  "duration_ms": 41,
  "complete": true
}

field
`id`	The request ID. The client got the same value in `x-amz-request-id`, and in the `RequestId` of an error body.
`api`	`s3`, `admin`, `console`, `sts`, `iceberg`, `unity`, `delta-sharing`
`action`	The action as policies name it (`s3:GetObject`, `s3:PutBucketPolicy`, `sts:AssumeRoleWithWebIdentity`). Absent for admin and console calls; read `method` and `path` instead.
`bucket`, `key`, `bucket_tenant`	What it acted on, and that bucket's tenant
`principal.auth`	`Permanent` (an access key), `Sts`, `AssumedRole`, `Anonymous`, `Console`, or absent when the request was refused before an identity was established (a bad signature, for example)
`source.ip`	The client, behind trusted proxies
`source.endpoint`	The named listener it arrived on
`status`, `error_code`	HTTP status and S3 error code (`AccessDenied`, `NoSuchKey`, …)
`request_bytes`, `response_bytes`	Bytes in (from `Content-Length`) and out (counted as sent)
`duration_ms`, `complete`	Time to the last byte, and whether the whole response went out. `false` means the client hung up mid-transfer.

Refused requests are events too: a denied anonymous read or a request with a wrong signature is recorded with its status and error code.

Never recorded: the Authorization header, and a presigned URL's X-Amz-Signature, X-Amz-Credential and X-Amz-Security-Token, which are replaced by ******** in query.

Where events go¶

A file on the gateway host¶

iolake-gateway --audit-log /var/log/objectio/audit.log   # or "-" for stdout

The file gets every event as JSON lines, and is reopened when log rotation moves it away. It's set only on the command line (or IOLAKE_AUDIT_LOG): a host path is not something an API caller should choose.

The operator's targets¶

PUT /_admin/audit
{
  "enabled": true,
  "include_reads": true,
  "targets": [
    {"type": "webhook", "name": "siem", "url": "https://siem.example/iolake",
     "auth_token": "…", "batch_size": 500, "flush_ms": 2000, "queue_size": 10000},
    {"type": "stdout", "name": "stdout"}
  ],
  "allowed_tenant_hosts": ["siem.acme.example", "*.logs.globex.example"]
}

webhook receives POSTs of application/x-ndjson (one event per line), with Authorization: Bearer <auth_token> when a token is set.
stdout writes to the gateway's standard output, for Kubernetes log collection.
include_reads: false drops GET and HEAD events, which are most of the volume on a read-heavy cluster.
allowed_tenant_hosts lists the hosts tenants may send their own events to (below).

Every gateway re-reads the configuration within 10 seconds; the one that took the change applies it before it answers, so a target added there gets the event of every request that gateway answers afterwards. GET /_admin/audit reads it back with tokens as ********. Writing that placeholder back keeps the stored token.

A tenant's targets¶

A tenant admin configures the same way, for its own tenant:

PUT /_admin/audit           # as acme's admin
{"targets": [{"type": "webhook", "name": "acme-siem",
              "url": "https://siem.acme.example/ingest", "auth_token": "…"}]}

A tenant receives the events on its buckets, including the operator's own requests on them, and by its principals. It never sees another tenant's.

A tenant's targets are webhooks only: a stdout target writes to the gateway's own output, so only the system admin sets one, in the cluster's configuration. A tenant configuration with any other type is refused with 400, saying so. A tenant's webhooks must be https and on a host the operator lists in allowed_tenant_hosts. Without that rule, a tenant could point the gateway at addresses inside the cluster, such as meta's port or a cloud metadata endpoint. Redirects are never followed, for the same reason. If the operator later removes a host, delivery to it stops.

Delivery¶

Delivery is at least once. A batch is retried with backoff (1 s up to 30 s) until the target takes it (a webhook answers 2xx). A target can therefore see an event twice, so deduplicate on id. Auditing never holds up or refuses a request.

With a spool (production)¶

With --audit-spool <dir>, every event is written to the gateway's disk before it is delivered:

Each event is written when its request completes, to the current segment in the spool (one write, so it survives the gateway being killed).
Only fsynced events are delivered. The spool is fsynced every 50 ms, so a power cut never leaves a target holding an event the spool lost.
Each target has a shipper that reads from its own cursor, kept in the spool directory. The cursor only moves once the target has taken a batch. A target that is down lets the spool grow; it is caught up when it's back, however long it was down, and so is a gateway restarted after being killed.
A segment every target has passed is deleted. A new target gets events from the moment it's added.
On shutdown (SIGTERM, as Kubernetes and systemd send, or Ctrl-C) the gateway delivers what's left before it exits, for up to --audit-drain-secs (default 20). On Kubernetes the pod's grace period must cover it (see how a node stops).
The spool is capped by --audit-spool-max-bytes (default 10 GiB). Past that, new events are dropped and counted under target="spool".

The system always audits. With --audit-system-bucket <name> (it needs the spool), every event also goes to that bucket in the cluster, which is created on first start with a lifecycle rule expiring events after --audit-retention-days (default 365; 0 keeps them). Events are stored as JSON lines, one object per batch, by hour:

<bucket>/2026/10/04/12/<gateway>-<segment>-<offset>.ndjson

Each object is named after its place in the spool, so a batch delivered again replaces itself.

Helm turns both on by default (gateway.audit in values.yaml). The spool is an emptyDir with a 10 GiB sizeLimit, and the gateway's own cap is 9 GiB, so the volume never fills. A container restarted (the gateway killed) keeps its spool. A pod that is deleted delivers its spool during its termination grace. A pod that is deleted without a shutdown loses what it hadn't delivered.

Without a spool (development)¶

Events wait in memory, in a queue per target (queue_size, default 10 000). If a receiver is down long enough to fill the queue, new events for it are dropped and counted. A gateway that crashes loses what it hadn't delivered.

metric
`iolake_audit_events_total{target}`	delivered (`target="system"`: the system bucket)
`iolake_audit_dropped_total{target}`	dropped: `spool` (the spool is full), `ingest` (no spool: the gateway's intake), or a target's full queue
`iolake_audit_delivery_failures_total{target}`	failed delivery attempts (each retried)

Alert on iolake_audit_dropped_total increasing.

SDKs¶

admin.put_audit_config({"targets": [{"type": "webhook", "name": "siem", "url": "https://siem.example/x"}],
                        "allowed_tenant_hosts": ["siem.acme.example"]})

_, err := admin.PutAuditConfig(ctx, "", iolake.AuditConfig{
    Targets: []iolake.AuditTarget{{Type: "webhook", Name: "siem", URL: "https://siem.example/x"}},
})