Failure Recovery¶

Handling failures in IOLake clusters.

Failure Domains¶

IOLake is designed to tolerate failures at multiple levels:

┌─────────────────────────────────────────────────────────────────┐
│                    Failure Domain Hierarchy                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Datacenter ──┬── Rack ──┬── Host ──┬── Disk                    │
│               │          │          │                           │
│               │          │          └── Single disk failure     │
│               │          │              (most common)           │
│               │          │                                      │
│               │          └── Node failure                       │
│               │              (power, OS, hardware)              │
│               │                                                 │
│               └── Rack failure                                  │
│                   (power, network switch)                       │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Disk Failures¶

Detection¶

OSDs detect disk failures via: - I/O errors during read/write - Scrubbing (background verification)

SMART readings are exported as metrics (iolake_disk_smart_*, where smartctl is installed) for you to alert on; nothing acts on them.

Automatic Recovery¶

With 4+2 erasure coding, the cluster tolerates 2 disk failures:

Object: photo.jpg (split into 6 shards)

Before failure:
  Disk1: shard1 ✓    Disk4: shard4 ✓
  Disk2: shard2 ✓    Disk5: shard5 ✓
  Disk3: shard3 ✓    Disk6: shard6 ✓

After Disk3 fails:
  Disk1: shard1 ✓    Disk4: shard4 ✓
  Disk2: shard2 ✓    Disk5: shard5 ✓
  Disk3: shard3 ✗    Disk6: shard6 ✓

Recovery: shard3 reconstructed from any 4 remaining shards
          → placed on spare disk (Disk7)

Replacing a disk¶

Lost shards are rebuilt automatically: peering on the meta leader finds what each placement group's members lack, and recovery rebuilds it. Watch it work (iolakectl metrics query needs the gateway's --prometheus-url; otherwise read the gateway's /metrics):

iolakectl pg list --state Degraded                        # PGs being made whole
iolakectl metrics query 'max(iolake_objects_at_risk)'   # objects one more loss from unreadable
iolakectl metrics query 'max(iolake_degraded_bytes)'
iolakectl metrics query 'rate(iolake_meta_pg_recovered_shards_total[5m])'
iolakectl metrics query 'max(iolake_meta_pg_objects_unfound)'

To take an OSD's disk out of service (still readable):

iolakectl node list                               # find the OSD's id
iolakectl osd set-state <osd-id> draining         # copies everything off it
iolakectl cluster drain-status                    # progress; it turns "out" when done

When a drain finishes, the OSD is marked out and wiped (its space is reclaimed, and nothing stale can come back with it). Replace the hardware, start the OSD on the new disk (a blank disk takes the place of the one missing), and set it back in:

iolakectl osd set-state <osd-id> in

For a disk or host that is gone for good, set its OSD out straight away:

iolakectl osd set-state <osd-id> out
iolakectl cluster drain-status        # progress of the evacuation

Everything it held is rebuilt elsewhere, whether or not it still answers: each shard from the rest of its stripe (or copied, if the OSD can still serve it), and each object's metadata copy, inline objects included, on the OSD that takes its place. When nothing refers to it any more it is purged. A drive that comes back, or its replacement, joins as a new OSD, or is set back in once the purge is done; don't put a directly-out OSD back in with old data on it.

An OSD with several disks doesn't start with one of them missing: its log names the disk, and it never runs on the rest, which would look for shards on the wrong disks. Name the disk again (any path, any order), or replace it with a blank one; with no replacement, set the OSD out and, once evacuated, start it afresh on the disks it has left (production runbook).

Full disks¶

Client writes may fill each OSD disk up to its full ratio (--full-ratio, default 0.95, as Ceph's full_ratio). Past that:

Writes are refused with 507 StorageFull when a shard's OSD has no room under its ratio and the write can't reach its quorum without it. S3 clients don't retry a 507 blindly, as they would a 503.
The rest of the disk is kept for writes that restore redundancy: repair, drain and backfill set use_reserve on their shard writes. A shard lost on a full cluster can still be rebuilt.
Nothing acknowledged is lost, and reads and deletes go on.
Deleting frees the space, and writes are admitted again.

The filesystem under an OSD filling is answered the same way. A file-backed disk is sparse, so its filesystem can run out before the disk reaches its ratio; the filesystem of the OSD's state directory, which holds its metadata, can fill too. A write the filesystem refuses for lack of room (ENOSPC, or EDQUOT on a quota) fails with 507 StorageFull, and nothing of it is kept. The OSD keeps serving reads, isn't reported unhealthy for it, and takes writes again as soon as there is room, without a restart. One exception: if an fsync of the OSD's metadata log fails, even for lack of room, the OSD takes no more writes until it is restarted, because nothing written since the last good sync can then be trusted to be on disk.

When some of a write's OSDs are full and others are unreachable, the answer depends on whether the unreachable ones could still make the quorum: if they could, it's 503 (a retry may succeed once they're back); if the full ones alone keep the write short, it's 507.

Watch for it before it happens. Alert when an OSD passes about 85%:

iolake_osd_used_bytes / iolake_osd_capacity_bytes > 0.85

Then add OSDs or a pool, or delete data. Tested by the full_disk e2e suite: a 4+2 cluster on 1 GiB disks is filled with 4 MiB objects until writes are refused. The refusals are StorageFull, no OSD passes about 95%, the acknowledged objects read back intact, and deleting half lets writes in again.

Meta's disk filling¶

Each meta node keeps a reserve free on the filesystem of its data directory: --min-free-mib, 1 GiB by default. Its database, Raft log and snapshots all live there.

Under the reserve on the leader, writes that need meta are refused with 507 StorageFull, its message naming the node's free space: creating a bucket, IAM changes, multipart uploads, settings. An object's PUT still succeeds, since its metadata is kept on its OSDs; it shows in listings once meta has room again.
Reads go on: GETs, listings, HEADs, the admin API's reads.
A follower takes its leader's entries down to half the reserve, then falls behind, and catches up once there is room. Writes go on while a majority has room; with a majority short they are refused, 507, as above.
Room back, writes resume within a second or two. Nothing is restarted and nothing acknowledged is lost.
A snapshot (every --raft-snapshot-every entries) is taken only with room for it above the reserve; until then the log isn't compacted and the node logs raft snapshot waits.

What to do: free space on that filesystem (nothing of meta's own is safe to delete by hand) or grow the volume. curl http://<meta>:9102/status shows disk_free, disk_reserve, and disk_full with the reason while the node is short; the metrics are iolake_meta_disk_free_bytes and iolake_meta_disk_full, and the chart's alerts IOLakeMetaDiskLow and IOLakeMetaDiskFull fire before and when it happens (production).

The reserve covers what meta writes after it stops taking writes. If something else on the same filesystem takes the last bytes too, a write of meta's own can still find it full. A Raft log write then waits, and is tried again every second (the node logs raft log: no room for an append): a follower falls behind, a leader stops answering and the others elect a new one. A write to meta's database that fails stops the node, which exits, and its supervisor starts it again: it applies its log again from its last checkpoint, and is back once there is room. Nothing acknowledged is lost either way. Keep meta's data directory on a volume of its own, so nothing else can fill it.

OSD Failures¶

Types of OSD Failures¶

Type	Detection	Impact
Process crash	Heartbeat timeout	Temporary unavailability
Network partition	Heartbeat timeout	Partial cluster access
Hardware failure	No heartbeat	Requires repair

Recovery Process¶

Meta probes every OSD every 15 s (every 2 s just after it starts); three misses in a row (about 45 s) mark it Down.
For 120 s after that (config pg/down_out_seconds), its placement groups keep it. Writes go on degraded while each stripe still reaches its write quorum on distinct OSDs (k + 1 shards, two copies, or all but g − 1 shards of an LRC stripe; 503 short of it), and reads decode around it. An OSD back within that time is brought up to date with what was written meanwhile; nothing moves.
After 120 s, each of its placement groups gets a stand-in OSD within the pool's placement rule, and recovery backfills the PG onto it from the rest of each stripe. A PG with no OSD to stand in is Undersized.
Setting it out (iolakectl osd set-state <node_id> out: the host or drive is gone for good) evacuates whatever still names it and then purges it, as above; don't put an OSD set out back in with its old data.

See the production runbooks.

How long repair takes¶

How long repair takes depends on how much there is to rebuild, not on how long the OSD was away:

An OSD back after a short outage (a restart, or a host off or cut off for a minute) gets only the writes it missed. With six OSDs and 4+2, one OSD was stopped or frozen for 60 s while about 2,300 objects were written. Every placement group was Clean 19–29 s after it was back. On the chaos VMs, cutting the meta leader's host (and its OSD) off the network for 60 s took 151 s, with writes going on throughout.
A replaced or lost drive has everything it held rebuilt. With six OSDs and 4+2 every placement group has a shard on every OSD, so that is a shard of nearly every object. On the chaos VMs a blank replacement disk took 482 s to fill, and a lost drive's 72,738 shards took 1,129 s to move to the OSD that replaced it. That is about 60–150 shards a second, at the pace of those VMs' disks and network.

Objects stay readable and writable throughout. Until a placement group is Clean again, some of its objects have less redundancy than the pool's scheme gives. To watch the rebuild, use the commands above: iolakectl pg list --state Degraded shows what is left, and iolake_meta_pg_recovery_objects_remaining and iolake_meta_pg_recovery_bytes_remaining show how much.

Each OSD takes part in at most two placement groups' recovery at once (config pg/backfills_per_osd), so that client I/O keeps priority. When every placement group has a member on every OSD, as with six OSDs and 4+2, that means two placement groups at a time across the whole cluster. On disks with I/O to spare, raise it to rebuild faster:

iolakectl config set pg/backfills_per_osd --value 4

pg/recover_at_once (8) caps how many placement groups the meta leader recovers at once, whatever each OSD allows.

Manual Recovery¶

iolakectl node list                  # status and admin state of every OSD
iolakectl node show <osd-id>         # its disks and capacity
iolakectl cluster topology           # where it sits in the failure domains
iolakectl metrics query 'iolake_meta_repair_last_pass_timestamp_seconds'

Repair is continuous; there is nothing to start by hand. An OSD that is down long enough to be replaced should be set out (see above).

Restarting a Failed OSD¶

# Check logs for failure reason
journalctl -u iolake-osd -n 100

# Common issues:
# - Disk I/O errors → check disk health
# - Out of memory → increase limits or add RAM
# - Network issues → check connectivity

# Restart OSD
sudo systemctl restart iolake-osd

# Verify reconnection (its node_id or name, from `iolakectl node list`)
iolakectl node show <osd-id>

Metadata Service Failures¶

Raft consensus (openraft 0.9)¶

The metadata service runs openraft. Every meta pod is a Raft member; cluster size is a deployment decision via meta.replicas in the helm chart:

`meta.replicas`	Semantics	Survives
1	Single-voter cluster	Pod restart with PVC intact
3	3-voter quorum	1 pod loss
5	5-voter quorum	2 pod losses

Every metadata change (buckets, users, keys, tenants, pools, placement groups, configuration, …) goes through the Raft log, so a new leader has it (what is replicated).

Admin API¶

All endpoints are on the meta admin port (default 9102), meta-local and not exposed via the gateway. Without mTLS they are plain HTTP and unauthenticated: keep the port to the network only IOLake and its operators reach. With mTLS the port is HTTPS, and every call below but /status takes a client certificate of role admin: add --cacert ca.pem --cert admin.pem --key admin.key and use https://.

# Cluster state (leader id, term, voters, learners, last_log_index)
curl http://meta-0:9102/status | jq

# Add a node to the cluster as a learner (non-voting, catches up
# via log replication); addr is its gRPC port:
curl -X POST http://meta-0:9102/add-learner \
  -H 'Content-Type: application/json' \
  -d '{"node_id": 4, "addr": "meta-3.meta-headless:9100"}'

# Promote learners to voters (pass the full voter list, not a delta):
curl -X POST http://meta-0:9102/change-membership \
  -H 'Content-Type: application/json' \
  -d '{"voters": [1,2,3,4]}'

Single pod failure in a 3-node cluster¶

The cluster continues with 2 of 3 nodes. A follower forwards every call to the leader, so clients see no difference while a follower is down. When the leader is lost, the followers stand for election after hearing nothing from it for 3–4 s (its 2 s lease and a random 1–2 s timeout, --raft-election-ms, 1000, with heartbeats every --raft-heartbeat-ms, 500); calls in flight fail with 503 to S3 clients, which retry, and writes resume once a new leader is elected. A leader answers only while a majority has confirmed it leads within the last half second, so one cut off from the others stops answering rather than serve stale state.

# Watch the others elect a new leader after the leader's pod is killed:
for i in 1 2; do
  curl -s http://<name>-meta-$i.<name>-meta-headless:9102/status
done

A pod that comes back on its volume reads its Raft log, finds the newer term, and rejoins as a follower without operator action. The multi-process end-to-end tests that cover this are listed in meta HA.

Advertise addresses matter. The membership stores each node's --raft-advertise address (the chart sets it to the pod's DNS name on the gRPC port, 9100). A node that advertises 0.0.0.0 is unreachable to its peers.

A meta node whose Raft stopped¶

A storage error while a meta node applies the log (a failing disk, a full volume) stops its Raft for good. The node logs Raft stopped: …; exiting so this node is restarted and exits with status 1; Kubernetes (the chart's liveness probe is meta's /health, which answers 503 meanwhile), systemd or aio restarts it, and it applies the log again from its last checkpoint. If it stops again at the same place, the cause is the node's storage: check its disk and its log.

A request that can never apply, such as a compare-and-set of more than 256 operations, is refused before it reaches the log (InvalidArgument; 400 through the gateway's APIs) and stops nothing. One that reached the log anyway (written by an earlier release) is applied as refused, writing nothing, on every node.

Losing a meta majority¶

Meta survives the loss of a minority of its nodes. A majority lost leaves the group without a leader: nothing is written until it is back.

The nodes are down but their data is intact (pods rescheduling, hosts rebooting, a partition): start them. Each reads its Raft log and the group elects a leader; nothing else is needed.

kubectl rollout restart statefulset/<name>-meta -n iolake

A majority's data is gone for good, and one node's is intact: make that node the only voter (below).
Every node's data is gone: restore the last backup (below). Without a backup the objects' bytes are still on the OSDs, but nothing names them: the cluster can't be recovered.

The OSDs keep the objects' bytes in either case. What a restore or a forced membership can't give back is meta's own record of what changed after the point it restores to.

Backing up meta¶

iolakectl meta export --file meta-$(date +%F).backup

The system admin takes a backup of a live cluster, as it serves: the leader, once a majority confirms it leads, writes every table of meta's state from one consistent read (buckets, object listings, users, keys, policies, tenants, pools and placement groups, packs, catalogs, block volumes, config). It holds every write acknowledged before it. iolakectl checks the file as it downloads (a SHA-256 over the whole of it), writes it to FILE.part and gives it its name only once it is whole: a transfer cut short fails and leaves nothing. Without --file it goes to standard output.

Take one on a schedule, and before every upgrade. A restore loses everything written to meta since its backup.
Keep it off the cluster, and private. It holds every secret meta keeps: users' secret keys, the cluster's signing secret, the KMS keys (wrapped by the SSE master key). iolakectl writes it readable by its owner only. The SSE master key isn't in it: keep that apart, as you already must.
A backup is about as large as meta's database: one listing entry per object version.

Restoring meta from a backup¶

When every meta node's data is lost. The OSDs must be the cluster's own, with their disks.

Stop every meta node, and empty its data directory. A node that starts on its old data would lead a group of its own beside the restored one. On Kubernetes, scale the meta StatefulSet to 0 and delete its volumes' contents (or the PVCs).
Restore into one node's empty data directory, with meta stopped, as the id and address that node starts with:

iolake-meta restore --from meta-2026-10-20.backup \
  --data-dir /var/lib/objectio/meta --node-id 1 \
  --raft-advertise meta-0.meta-headless:9100

It refuses a data directory that holds a meta node's data, a file that isn't a whole backup (checked end to end before anything is installed), and a backup of a cluster at a format level this binary can't run at (a newer release's, or one older than the oldest it upgrades from). It prints what the backup is (cluster, release, level, Raft index) and what to check after. A restore that stops part way leaves restore-incomplete in the directory, and meta refuses to start on it: empty the directory and restore again. 3. Start meta on it. It is a group of one voter, and elects itself. 4. Add the other nodes, empty, as when the group was formed: /add-learner for each, then /change-membership with every voter (the admin API).

What the restored cluster holds, and what it doesn't:

Objects written after the backup are listed again. Meta lists every placement group once, from the OSDs, and adds the objects it finds; ones deleted after the backup are unlisted. Until it has listed them all (iolake_meta_pgs_to_relist falls to 0), a listing may miss them; a GET of one by name reads it.
Everything else meta keeps is as it was at the backup: buckets, users, keys, policies, tenants, pools and config made after it are gone, and ones deleted after it are back. Block volumes' writes after it are lost.
Objects packed after the backup can't be read: where a pack keeps each object is recorded only in meta.
Space is held, never lost: a stripe made before the restore is not freed when its object is deleted, since a copy or pack made after the backup may still use it (iolake_meta_restore_frees_held_total).
OSDs the backup held as draining or out are left alone: one may have been put back in since, with data on it. Nothing drains or purges it until you set its state as it really is, iolakectl osd set-state ID in|out|draining (iolake_meta_restore_osds_unconfirmed; the restore names them).
Every placement group moves to a new epoch, above any the lost group gave the OSDs, so the OSDs take the restored meta's placements and refuse the old ones.

The last resort: one survivor leads alone¶

When a majority of meta's voters is lost for good (disks destroyed, volumes deleted) and one node's data is intact, that node can be made the only voter of its group, without a backup.

This is dangerous. A lost node that comes back with its data still belongs to the old group, and two of them would elect a leader of their own beside the survivor: two groups giving the OSDs different answers. Use it only when a majority can't be started again:

Make sure no other voter can ever start on its data: wipe its data directory or delete its volume. On Kubernetes, scale the StatefulSet to 0 and delete the lost pods' PVCs.
With the survivor stopped, make it the only voter, saying the others are gone:

iolake-meta force-single-voter --data-dir /var/lib/objectio/meta \
  --node-id 2 --others-lost-for-good

Without --others-lost-for-good it refuses and changes nothing. --raft-advertise gives its address if it changed; by default it keeps the one its group knew. 3. Start it, then add new nodes with empty data directories, as after a restore.

The survivor applies every entry in its log, including ones past the commit point it knew, which may never have been acknowledged. Entries the lost majority committed that never reached it are gone; it prints how far it had applied, how far its log went and the last commit point it knew. It then behaves as a restored cluster does (above): placement groups listed again and moved to a new epoch, frees of older stripes held, draining or out OSDs left for you to confirm. It loses less than a restore from a backup, which loses everything since the backup.

Network partitions¶

Raft's quorum requirement prevents split-brain. In a 3-node cluster partitioned 1 + 2:

Minority side (1 pod): can't elect a leader; calls to it fail (UNAVAILABLE, 503 to S3 clients) and clients go to the other nodes.
Majority side (2 pods): re-elects a leader if the old one was on the minority side; continues serving writes.

When the partition heals, the minority catches up via AppendEntries and resumes as follower. No manual intervention needed.

Gateway Failures¶

Stateless Recovery¶

Gateways are stateless - simply restart:

sudo systemctl restart iolake-gateway

Load Balancer Health Checks¶

Check gateways at GET /_ready: it answers 503 until the gateway can serve, and again while it shuts down, so a load balancer stops sending it requests first. GET /health only says the process is up (health endpoints).

# nginx: passive checks (active /_ready checks need nginx Plus or another LB)
upstream iolake {
    server gateway1:9000 max_fails=3 fail_timeout=30s;
    server gateway2:9000 max_fails=3 fail_timeout=30s;
}

Network Partitions¶

Meta-side partition behavior is covered in the Metadata Service Failures section above. Gateway/OSD partitions are handled by client-side retry.

Client Retry Logic¶

Clients should retry on different gateways:

import boto3
from botocore.config import Config

# Configure retries
config = Config(
    retries={
        'max_attempts': 3,
        'mode': 'adaptive'
    }
)

s3 = boto3.client('s3', config=config)

Data Corruption¶

Detection¶

Online: CRC32C verification on every read
Offline: Background scrubbing

Recovery¶

A read that finds a shard failing its checksum decodes the object from the other shards and returns correct data; scrub finds the rest, and recovery rebuilds the bad shard. There is no manual step. Watch iolake_osd_corrupt_shards (back to 0 once rebuilt), iolake_meta_repair_unrecoverable_stripes_total and iolake_meta_lost_objects (should stay 0).

Disaster Recovery¶

Backups¶

Meta is a Raft group (3+ nodes) and survives the loss of a minority. Against losing every meta node's data, back meta up with iolakectl meta export on a schedule and before every upgrade, and keep the file off the cluster. To protect against losing the whole cluster, replicate buckets to another IOLake cluster with bucket replication (asynchronous, per bucket), or copy them with an S3 tool (rclone sync, aws s3 sync) on a schedule.

Recovery Time Objectives¶

Failure Type	RTO	RPO
Single disk	Automatic	0 (no data loss)
Single OSD	Automatic	0 (no data loss)
Single metadata node	Seconds (an election)	0
Loss of a meta majority, data intact	Until a majority is back	0 (Raft log)
Loss of a meta majority's data, one survivor	Forced membership, then an election	What the survivor hadn't received
Loss of every meta node's data	Restore, then the PGs listed again	Meta's changes since the last backup (objects are listed again)
Complete cluster loss	Fail over to the replica, or restore from your copy	Replication lag, or the last copy

Failure Simulation¶

Test recovery in a staging cluster by failing things for real: stop an OSD process (systemctl stop iolake-osd), detach a disk, or partition a meta node with a firewall rule. Then check that iolake_objects_at_risk returns to 0 and that a checksum-verified read of a sample of objects (aws s3 cp and compare) succeeds. The end-to-end tests in the source tree (tests/e2e/tests/{repair,drain,pg_recovery}.rs, and meta_ha.rs for meta) do this against real processes.