Bucket replication

Copy every new version written to a bucket into a bucket on another IOLake cluster, asynchronously, so losing a site loses nothing that had time to replicate.

Set it up

1. On the target cluster: a versioned bucket, and credentials for the source to write with. The admin's work; for least privilege, a user with s3:PutObject, s3:ReplicateObject (and s3:DeleteObject, s3:ReplicateDelete if delete markers replicate) on that bucket.

aws s3api create-bucket --bucket dr-copy --endpoint-url https://site-b:9000
aws s3api put-bucket-versioning --bucket dr-copy \
  --versioning-configuration Status=Enabled --endpoint-url https://site-b:9000

2. On the source cluster: the target. Targets belong to a tenant: a bucket's rules can name only its own tenant's targets, so one tenant can never send data into another's remote bucket with the other's credentials. A tenant's admin manages its tenant's targets; the system admin manages the system's (no ?tenant=) and any tenant's (?tenant=).

curl -X POST https://site-a:9000/_admin/replication/targets \
  --aws-sigv4 aws:amz:us-east-1:s3 --user "$AK:$SK" \
  -H 'content-type: application/json' \
  -d '{"name": "site-b", "endpoint": "https://site-b:9000", "bucket": "dr-copy",
       "access_key": "...", "secret_key": "...", "bandwidth": 104857600}'

Or with iolakectl (the remote secret from IOLAKECTL_REMOTE_SECRET_KEY if not given, which keeps it out of the process list):

iolakectl replication target set site-b --remote-endpoint https://site-b:9000 \
  --remote-bucket dr-copy --remote-access-key ... --bandwidth 100M
iolakectl replication target list

bandwidth is optional: the bytes a second the whole cluster may send to the target (see Bandwidth); without it, as fast as the link goes.

GET /_admin/replication/targets lists a tenant's targets (the caller's own, for a tenant admin; secrets never shown); DELETE /_admin/replication/targets/{name} removes one. Both take ?tenant= for the system admin. The secret is kept in meta, as users' secret keys are.

A target a tenant's admin sets must be https://, on a host the operator allows tenants (as for tenant audit webhooks); otherwise a tenant could make the gateway send requests to addresses inside the cluster. Until the operator allows some, tenant admins can't set targets, and the system admin sets them with ?tenant=, unrestricted:

curl -X PUT https://site-a:9000/_admin/replication/settings \
  --aws-sigv4 aws:amz:us-east-1:s3 --user "$AK:$SK" \
  -H 'content-type: application/json' \
  -d '{"allowed_tenant_hosts": ["dr.example.com", "*.dr.example.com"]}'

Or with iolakectl (settings set replaces the list; with no host, tenant admins can't set targets):

iolakectl replication settings set \
  --allowed-tenant-host dr.example.com --allowed-tenant-host '*.dr.example.com'
iolakectl replication settings get

The list is checked again at every send, so removing a host stops tenants' targets on it.

3. On the source bucket: versioning, then the rules (S3's PutBucketReplication), by the bucket's owner or anyone its policy grants s3:PutReplicationConfiguration. The destination names one of the bucket's tenant's targets: arn:obio:replication:::<target>.

<ReplicationConfiguration>
  <Rule>
    <ID>everything</ID>
    <Status>Enabled</Status>
    <Filter><Prefix></Prefix></Filter>
    <DeleteMarkerReplication><Status>Disabled</Status></DeleteMarkerReplication>
    <Destination><Bucket>arn:obio:replication:::site-b</Bucket></Destination>
  </Rule>
</ReplicationConfiguration>

aws s3api put-bucket-replication --bucket photos \
  --replication-configuration file://rules.json --endpoint-url https://site-a:9000

A rule filters by prefix and tags (<And> for both). Both buckets must be versioned; the rules are refused otherwise.

What replicates

Change on the source	On the target
A new version (PUT, multipart upload, copy)	The same version: same version id, bytes, ETag, content headers, x-amz-meta-*, tags, Object Lock retention and legal hold, SSE-S3 / SSE-KMS encryption (re-encrypted with the target's keys; an SSE-KMS key id must exist there)
A delete marker	Only if the rule has DeleteMarkerReplication Enabled
A DELETE of a specific version	Never: a mistake or an attacker on one site can't destroy the other's copy
An SSE-C object	Not replicated (its key isn't ours to send), as in S3
Objects written before the rule	Only through a resync

The target never replicates a replica onward, so two clusters with rules pointing at each other don't loop.

How it works

• A version a rule covers is committed with each target marked PENDING, in the same metadata write: no version is ever committed and forgotten.
• The gateway that took the write sends it at once. One gateway at a time (a lease in meta) also scans every bucket with rules, every --replication-scan-secs (30 s by default), for versions still PENDING or FAILED, and sends them oldest first. That is what makes it reliable: after a crash, or while a target was down, every marked version is found.
• A failed send is retried with backoff (2 s, doubling, up to 5 min), indefinitely. FAILED is a status to alert on, not a give-up.
• Objects up to 64 MiB go in one PUT; larger ones as a multipart upload of 16 MiB parts.
• The target stores a version it already holds (same version id and ETag) once: a send repeated after a crash changes nothing. An older version arriving after a newer one is kept as a version, never made current.

Existing objects: resync

A rule sends what is written after it. To send what a bucket already holds (turning replication on for a bucket with data, or bringing a new or rebuilt target up to date), start a resync of the bucket to one of its targets. The bucket's tenant's admin, or the system admin, may:

iolakectl replication resync start photos site-b
# or: POST /_admin/replication/resync {"bucket": "photos", "target": "site-b"}

The bucket must have an enabled rule sending to the target. The scanner (one gateway at a time) walks the bucket in key order, a page of 1000 keys at a time: every version a rule sends to the target and that the target hasn't got is marked PENDING and sent. Up to 8 sends run at once, within the target's bandwidth limit. Each pass gives resyncs a minute, then scans for versions not yet sent as usual, so a resync of a big bucket doesn't hold back other buckets' retries.

A version already COMPLETED for the target is left alone, and so is one PENDING or FAILED (the fast path and the scanner send it). To send everything again, for a target that was replaced or lost data, add --again ("again": true): versions already sent are marked and sent too; the target keeps one copy of each.

Watching and stopping it:

iolakectl replication resync status photos
# TARGET  STATE    AFTER        SCANNED  MARKED  SENT    FAILED  BYTES     ERROR
# site-b  running  img/0412.jpg 412000   409870  409870  0       88123456  -
iolakectl replication resync cancel photos site-b

GET /_admin/replication/resync?bucket= gives each target's resync (running, or the last one, done or cancelled): the last key done (after), the versions looked at (scanned), those a rule sends to the target (covered), marked, sent, failed, the bytes, and the last error. Starting a resync that is running returns it as it is.

A resync survives restarts. Its record in meta moves past a page only once every version in it is marked and sent. If the gateway stops mid-page, the scanner, wherever it runs next, takes the page again (after the scanner's lease runs out, up to a minute or three scan intervals); what that page had marked but not sent is PENDING by then and goes out with the scanner's next pass. Cancelling keeps what was sent and marked; the marked versions still go.

Bandwidth

A target's bandwidth is the cluster's limit for it: the fast path, the scanner and resyncs on every gateway together send it no more than that many bytes a second. Each gateway takes an equal share (the limit divided by the gateways up, as meta last heard from them, refreshed every 30 s) and paces its sends to it, 64 KiB at a time; a quiet spell saves nothing up, so no burst follows it. A gateway can't use another's idle share, so with traffic on one gateway only, the cluster sends below the limit, not above it; and for the seconds before a new gateway first reports, the others still use the old shares.

The limit counts request bodies (the objects' bytes), not headers. A paced send may take longer than the 5-minute send timeout; its timeout grows with what its bytes take at the gateway's share.

Watching it

• x-amz-replication-status on HEAD and GET: PENDING, COMPLETED or FAILED on the source, REPLICA on the target.
• Metrics (gateway /metrics):
• iolake_replication_backlog{target}: versions not yet replicated, at the last scan;
• iolake_replication_lag_seconds{target}: the oldest one's age, in effect the RPO;
• iolake_replication_sent_total{target,result};
• iolake_replication_bytes_total{target}.

Alert on lag that keeps growing, and on result="error".

Not yet

Two-way replication's conflict tests, and targets that aren't IOLake (AWS S3, MinIO), are the next phases.