Chaos: faults on a multi-host cluster¶
IOLake is tested under faults on a small cluster of separate machines
(deploy/chaos/ in the code repo): six VMs on one lab host. Three run meta,
all six run an OSD on its own block device, and two run a gateway. Six
writers keep PUTting through both gateways throughout. Each fault is
followed by a check:
- every write that was acknowledged reads back intact: the new ones plus a sample of 2,000 after each fault, and all of them at the end;
- writes resume, and fail only with 503 (retry) or a client timeout, never another error;
- at the end, repair restores full redundancy: once it is quiet, any of the listed pairs of OSDs can be stopped and every object still reads.
Latest: 2026-10-18: passes¶
The code of the 0.1.0 release candidate; 4+2 erasure coding; the VMs run release builds. A fault that destroys a drive waits first for every placement group to be Clean.
| Fault | What | Longest pause in acknowledged writes |
|---|---|---|
| Meta leader killed | SIGKILL of the meta leader |
2.2 s |
| OSD host powered off | one VM off for 60 s | 5.0 s |
| Meta leader's host powered off | a meta VM off for 60 s | 4.9 s |
| Partition | the meta leader's VM off the network for 60 s | 13.2 s |
| Disk pull | an OSD's disk removed for 60 s, then replaced as documented | 1.4 s |
| Drive lost | an OSD's drive and its metadata gone, the OSD back new on a blank drive, the old one evacuated | 1.0 s |
- Acknowledged objects: 81,436. All read back after every fault and at the end.
- Failed writes: 503s and client timeouts only.
- Repair: every placement group was Clean 151 s after the partition healed (before the disk pull) and 482 s after the new disk was set in (before the drive loss). The lost drive's OSD was evacuated in 1,129 s, 72,738 shards onto the new one.
- Redundancy: repair was quiet a minute after the last fault. Then any two OSDs (chaos-1+2, 3+4, 5+6) could be stopped with every sampled object readable.
The partition's pause is longer than on 2026-10-04 (5.4 s), within the 60 s the suite allows, and the same in both runs on this build. Its failed writes were client timeouts and 503s saying an election was under way; where the rest of the time goes isn't measured yet.
The run before it, on the same build, lost 247 objects. The faults then followed each other once every service was up, not once repair was done, and the drive loss (added to the suite with the soak) came 44 s after the disk pull. Repair was still rebuilding what chaos-3's OSD had missed while it was off (with six OSDs and 4+2, every placement group has a shard on every OSD), and every object lost had its shards on chaos-3, chaos-5 and chaos-4 gone: three of a 4+2 stripe, beyond what the code survives. The suite now makes disk-pull and drive-lost wait for every placement group to be Clean first.
2026-10-04: passes¶
A development build (with backfill and the last two fixes in the table below), the first five faults: the longest pauses 2.2 s (meta leader killed), 5.2 s (OSD host off), 5.3 s (meta leader's host off), 5.4 s (partition), 0.3 s (disk pull); 16,597 acknowledged objects, all read back; repair quiet 16 minutes after the last fault.
What the runs found¶
It took 15 runs to get here. Each run stops at the first broken invariant, so every fix uncovered the next problem. These are the problems the runs found and the fixes for them:
| Found | Fix |
|---|---|
| A follower meta never learned an OSD's admin state, so after a leader change an OSD set back in reverted to out | The admin state reaches every meta node |
| A gateway kept using connections to a host that had vanished: writes stalled for up to 48 s | Fail-fast per OSD address, HTTP/2 keepalive, stale channels redialled |
| Acknowledged writes lost to an OSD power cut | Compacting the OSD's metadata log no longer drops records, and syncs reach the live file |
| An OSD on a replacement disk took 5 minutes to start | The old disk's shard records are dropped in batches |
| Objects written while an OSD was down kept two shards on one OSD | A down OSD keeps its positions |
| An update (tagging, a pack switch) racing a PUT could outrank it, so the acknowledged PUT was lost | An update is ordered after its object, never above a newer one |
| A connection cut mid-answer became a 500 | It is a 503 |
| Objects written while an OSD was out kept two shards on one OSD | Backfill |
| A gateway failed fast OSDs it had just seen restart | It probes them until they answer |
| The redundancy check stopped OSDs while repair was still working | It waits for repair to be quiet |
| A request dropped by its client hid a dead connection, so a partition held one gateway's writes for 60 s | A slow call checks its OSD, and a dead OSD stays failed fast while it is gone |
| A healed partition deposed the leader: the cut-off meta had raised its term | A meta node campaigns only while it can reach a majority |
The soak¶
The soak runs the same faults, plus a drive lost for good, for 48 hours while twelve writers fill the cluster past the OSDs' full ratio and delete it back down. It passed for the first time on 2026-10-14, run 29: 6.37 M acknowledged operations, 1.37 TB written, 91 faults, nothing lost, and no space left with every key deleted. It passed again on 2026-10-18, run 38 on the 0.1.0 release candidate: 7.02 M acknowledged operations, 1.50 TB written, 84 faults (14 of each kind), nothing lost or leaked. The two 48-hour runs before it on the candidate's way there stopped at an OSD 305 s to serve after a power cut (run 36) and a meta node that wouldn't start after one (run 37). Every 48-hour run before it stopped at the first bug it met, or was stopped by hand to take a fix; each bug got a fix and a regression test.
A fix goes through the 3-hour quick soak first, then the 48-hour one.
deploy/chaos/run-soak.sh (its README) runs either on the lab host for
any commit at origin. It builds the commit in a clone of its own and
brings the cluster up on it, then records the run: logs and a
summary.json with faults by kind, the leak check and the verdict, plus
one line in runs.tsv.