Performance¶
IOLake next to RustFS (and SeaweedFS on one host), measured with the same client on the same machines and drives: a single-host baseline (again on 2026-10-15, after the small-object path), and six cloud VMs with local NVMe. Every change is checked against the tree it lands on by a regression check in CI. What a drive's write cache costs, with and without power-loss protection, is under drives. Where the CPU goes is in profiling.
The regression check in CI¶
The performance job in the code repo's CI runs in the merge queue, so
every change is measured before it lands; it is a required check on
main, so a change that fails it doesn't merge. It also runs on a pull
request labelled
perf (add the label, then push: a label alone doesn't start a run). It
builds the tree that lands and the tree it lands on (the base), both
release builds with ISA-L and io_uring, and runs
scripts/bench/regression.py on them.
What it runs. Each build five times, in turn on the same runner:
base, head, head, base, and so on. Every run starts a fresh
iolake-aio (six OSDs, 4+2) and, with iolake-s3-bench, puts 6,000
objects of 64 KiB (32 in flight) and gets each of them four times, then
puts 400 objects of 4 MiB (8 in flight) and gets each three times. A short
unmeasured warm-up comes first.
How it decides. For each of the four cells (PUT and GET, 64 KiB and 4 MiB), the head's median throughput is divided by the base's. Any cell below 0.85 fails the job. p99 latency is shown next to it but not judged: on a shared runner it moves too much from run to run.
Why it is built this way.
- Relative, on one runner. GitHub's runners differ from one another by more than most regressions, so no absolute number can gate anything. Both builds run on the same runner in the same job, and alternating them means a runner that slows down part way through slows both.
- A dedicated runner in the merge queue. The merge queue's run goes to
a self-hosted runner kept for this job, which moves less from run to run
than GitHub's shared ones. Each job gets a fresh container that is
thrown away afterwards, with no reach into its host's network, and the
runner takes only the merge queue's runs, never a pull request's. A
pull request labelled
perfruns on GitHub's runners, so its numbers are noisier. - Medians of five. One run in five can be off by a quarter (a cold first run, a noisy neighbour); the median ignores it.
- Data in memory. The aio keeps its data on a tmpfs. On a runner's disk, one build's 64 KiB PUTs went from 208 to 372 a second between runs, more than any regression this is for. In memory the check measures the software (CPU, copies, RPCs, locks), which is what changes from commit to commit.
What it doesn't see. A change that adds work on the drive (another fsync per PUT) costs almost nothing in memory, so it passes here. Drive costs are measured by the baselines below, on real drives, and by the soak.
Reading it. The job summary has a table:
| Cell | Base | Head | Head ÷ base | p99 (not judged) |
|---|---|---|---|---|
| PUT 64KiB | 1,237 ops/s (±1%) | 1,235 ops/s (±7%) | 1.00 ✅ | 85 → 88 ms |
The ± is half the range of the five runs. A wide range on either side
means that cell was noisy on this runner. The artifact performance has
every run's numbers (perf.json) and each aio's log. To run it locally:
cargo build --release --features iolake-aio/io-uring \
--bin iolake-aio --bin iolake-s3-bench # on each tree, into two directories
scripts/bench/regression.py --base <dir> --head <dir> --workdir <a tmpfs>
The work directory must not be under /dev: an OSD takes a path there
for a device.
How well it works.
Runs on GitHub's runners (a pull request with no change, and a throwaway pull request that added 1 ms of CPU to every PUT, about 30% of a 64 KiB PUT's CPU on a runner and almost none of a 4 MiB PUT's):
| Run | PUT 64KiB | GET 64KiB | PUT 4MiB | GET 4MiB | Verdict |
|---|---|---|---|---|---|
| No change | 1.01 | 1.00 | 1.05 | 1.02 | pass |
| No change, another runner | 1.01 | 1.01 | 1.08 | 0.99 | pass |
| 1 ms more CPU in every PUT | 0.78 | 1.00 | 0.95 | 0.96 | fail: PUT 64KiB |
The two "no change" runners differed by 65% in absolute speed (1,151 against 1,906 PUTs of 64 KiB a second); the ratios stayed within 1%. Getting there:
- On the runner's disk, the first version caught a 15 ms sleep in every PUT (PUT 64 KiB at 0.83), but on that runner the base alone ranged from 208 to 372 PUTs a second. Its data moved to a tmpfs.
- With a 4 MiB phase of 1.5 s, one no-change run came out at 0.87 for PUT 4 MiB, too close to the threshold. The phase now writes 400 objects instead of 150.
So it fails a change that costs a cell about 20% of its throughput or more; a smaller loss usually passes, so it is no substitute for the baselines. 4 MiB PUTs are the noisiest cell (one runner's runs spread ±24%).
Drives with and without power-loss protection¶
An acknowledged PUT waits for a flush on every OSD it writes to. On a drive with a volatile write cache and no power-loss protection (PLP), a flush empties that cache to flash; on one with PLP, capacitors cover the cache, so the drive can acknowledge a flush without emptying it. Small PUTs, which are mostly flushes, gain the most.
Measured: the drives we have, both without PLP. The lab host of the
single-host baseline has two consumer NVMe drives. Both report a volatile
write cache, and neither has PLP. 4 KiB writes, each followed by
fdatasync, one at a time (3,000 of them, 2026-10-15; other tests were
running on the host):
| Drive | Volatile write cache | Flush p50 | Flush p99 | Flushes a second |
|---|---|---|---|---|
| Corsair MP600 Elite 2 TB (the baselines' drive) | yes | 0.67 ms | 0.89 ms | 1,440 |
| OEM 1 TB, Phison controller (ESL01TBTLCZ; through LVM) | yes | 0.25 ms | 0.58 ms | 3,670 |
The six cloud VMs' local NVMe was not checked for a volatile cache while they ran, and they are gone.
Measured: a drive with no volatile cache, and a cloud disk. One Azure
Standard_L8s_v4 VM (Intel Xeon Platinum 8573C, 8 vCPUs, 64 GB, Linux
6.17), 2026-10-15, for an hour:
- Its local NVMe ("Microsoft NVMe Direct Disk v2") reports
vwc 0: no volatile write cache, and the kernel marks itwrite through, so a write is durable when it completes and anfdatasyncsends no flush. That is what a drive with PLP gives an fsync, however the drive gets there. - A Premium SSD v2 disk (64 GiB, its baseline 3,000 IOPS and 125 MB/s) is network storage: it reports a write-back cache with FUA, so flushes go through and each one is a round trip to the storage service.
The same fio flush test as above, three runs of 3,000 each:
| Drive | Volatile write cache | Flush p50 | Flush p99 | Flushes a second |
|---|---|---|---|---|
| Azure L8s_v4 local NVMe | no (vwc 0) |
0.11 ms | 0.20–0.22 ms | 7,300–8,400 |
| Azure Premium SSD v2, 64 GiB | write-back, flushed | 1.97 ms | 4.5–4.7 ms | 450–465 |
Then the small-object cells of the single-host baseline (warp, 32
clients, 30 s; scripts/bench/baseline.sh with SIZES="4KiB 64KiB"), on
one 4+2 iolake-aio built from main of 2026-10-15 (isal,
io-uring), its six OSDs' files on the drive under test. Two rounds, each
drive in turn; throughput (PUT p99):
| Cell | Local NVMe, no write cache | Premium SSD v2 |
|---|---|---|
| PUT 4 KiB | 1,547–1,571 obj/s (66 ms) | 333 obj/s (166–172 ms) |
| PUT 64 KiB | 717–756 obj/s (104–119 ms) | 270–271 obj/s (1.8–2.0 s) |
| GET 4 KiB | 8,694–8,704 obj/s | 8,840–8,849 obj/s |
| GET 64 KiB | 6,350–6,362 obj/s | 6,370–6,381 obj/s |
- Durable small PUTs follow the flush. Where a flush costs 0.1 ms, 4 KiB PUTs run 4.7 times as fast as on the network disk, where it costs 2 ms. The Premium SSD v2's 64 KiB PUTs also meet its baseline IOPS and bandwidth caps (six OSDs' writes share one disk); a production cloud disk would be provisioned past them.
- GETs don't care: the objects were just written and are in the page cache, so both drives read the same.
- Not the same machine as the lab host, so the absolute numbers aren't comparable with its tables; the flush cost, 0.11 ms here against 0.25–0.67 ms on the lab host's two consumer drives, is.
The operations guide already requires PLP (or a UPS) for production: without it, a power cut can lose what the cache held, whatever the speed.
Single host: 2026-10-15¶
The single-host baseline again, after the small-object path (the goal: small GETs within 1.5 times RustFS, 64 KiB PUTs level with it), on the same host and drive as 2026-10-04 below.
Systems: IOLake with the small-object path's last part
(iolake-aio, release, isal and io-uring, 4+2, six OSDs on 64 GiB
sparse files), the build before that part (main of 2026-10-14), and RustFS (rustfs/rustfs:latest of
2026-10-03, EC:2 over six directories). RustFS syncs before it answers a
PUT: strace of its PUT cells showed fsync and fdatasync throughout
(about 29 a PUT, which ext4 folds into about 2.4 device flushes).
Client: warp 1.1.4, 32 concurrent clients, 30 s per cell, PUT then GET for each size. Three rounds, the three systems in turn in each. Other work (builds and tests of other branches) ran on the host meanwhile, at a load of 8–13 on its 32 threads: read the ratios, not the absolutes.
Objects per second (p99 latency):
| Cell | IOLake | RustFS | IOLake before | RustFS ÷ IOLake |
|---|---|---|---|---|
| PUT 4 KiB | 1,146–1,300 (44–47 ms) | 285–341 (144–201 ms) | 205–316 | 0.24–0.27 |
| PUT 64 KiB | 988–1,096 (121–142 ms) | 302–371 (140–152 ms) | 162–318 (1.0–1.5 s) | 0.30–0.34 |
| GET 4 KiB | 25,854–27,044 (2.2 ms) | 31,473–32,177 (2.3–2.7 ms) | 17,643–17,713 | 1.16–1.24 |
| GET 64 KiB | 17,608–18,191 (3.0–3.5 ms) | 25,270–25,823 (2.7–2.9 ms) | 12,357–12,554 | 1.39–1.47 |
RustFS's PUTs depend on the state of the disk more than IOLake's: the same cells reached 859–880 64 KiB PUTs a second earlier in the day (when IOLake before did 337–719). IOLake's are above that too.
What changed (small-object path, part 5):
- Meta commits writes together. It used to take one write per Raft
log sync, and a 64 KiB PUT spent 95–186 ms waiting for
CreateObject. - An OSD's index values fit its pages. A 16 KiB small shard took a 32 KiB page of the index file; checkpoints wrote twice their bytes and held PUTs for seconds.
- A gateway reads an OSD's metadata several to a call, and the parity copies send no shard: CPU per 4 KiB GET fell from 1,020 µs to 463–524 (RustFS: 575–593).
All three need format level 8.
Six cloud VMs: 2026-10-07¶
Machines: six Azure Standard_D4pds_v6 in Central India, made with
iolake-infra (./infra up azure): 4 Arm vCPUs (Cobalt 100), 16 GB,
one local NVMe drive (220 GB) each, accelerated networking, one private
network. Ubuntu 24.04.
Systems, each on the same six drives in turn (the drives erased in between):
| Version | Layout | Durability of an acknowledged PUT | |
|---|---|---|---|
| IOLake | main of 2026-10-07 with the two profiling fixes below, release build, aarch64 (erasure coding by rust-simd: ISA-L is x86 only) |
meta on nodes 1–3, an OSD and a gateway on every node, 4+2. Each drive: 16 GiB for meta, 32 GiB for the OSD's metadata, the rest raw for the OSD | shards and metadata at write quorum, fsynced |
| RustFS | 1.0.1, aarch64 | one server per node, one pool of six drives, XFS, EC:2 (4+2; checked: a 4 MiB object puts 1,048,704 bytes on each drive) |
erasure-coded at write quorum, fsynced |
Meta shares each drive with the OSD here (these VMs have one fast drive); the guide gives it a drive of its own, and the p99 of PUTs pays for it (see below).
Client: warp 1.1.4 on node 1, spreading requests over all six nodes'
S3 ports, 60 s per cell, 64 concurrent requests (32 for 4 MiB). The
scripts are iolake-infra's deploy and bench.
Results¶
Throughput (median, p99 latency).
| Cell | IOLake | RustFS | IOLake ÷ RustFS |
|---|---|---|---|
| PUT 4 KiB | 1,921 obj/s (29 ms, 80 ms) | 291–382 obj/s (148–203 ms, 0.8–1.1 s) | 5–6.6 |
| PUT 64 KiB | 1,496 obj/s (33 ms, 341 ms) | 348 obj/s (139 ms, 812 ms) | 4.3 |
| PUT 4 MiB | 966 MiB/s (125 ms, 383 ms) | 787 MiB/s (150 ms, 303 ms) | 1.2 |
| GET 4 KiB | 8,316 obj/s (8.4 ms, 26 ms) | 4,532 obj/s (13.3 ms, 50 ms) | 1.8 |
| GET 64 KiB | 6,334 obj/s (10.9 ms, 29 ms) | 4,205 obj/s (14.0 ms, 54 ms) | 1.5 |
| GET 4 MiB | 2,759 MiB/s (43 ms, 115 ms) | 2,544 MiB/s (45 ms, 125 ms) | 1.1 |
What it shows¶
- Small objects: IOLake is ahead, on PUT by 4–6 times. A small PUT is one round trip to the OSDs and one flush on each: the shards go with the object's metadata, into the OSDs' metadata logs, which batch concurrent writes into one fsync.
- Small GETs: 1.5–1.8 times RustFS, where the single host had RustFS 3–4.5 times ahead: the GET is one round trip, and the single host's loopback RPC costs are gone.
- Large objects: within 1.2 times, IOLake ahead. Both are bound by the network and the drives.
- The p99 of small PUTs (341 ms at 64 KiB) is meta sharing its drive with the OSD: an OSD's metadata checkpoint holds back meta's syncs (see the hardware guide). The median is 33 ms.
Found along the way and fixed (details in profiling):
- A regex compiled for every request's
Authorizationheader: 27% of a gateway's CPU. - Every gateway's poll made every OSD scan all its object metadata: 20% of an OSD's CPU, and growing with the objects held.
- Meta killed in its first seconds could never start again; an OSD given
a missing
/devpath ran on a file in memory.
Single host: 2026-10-04¶
Host: one lab machine. AMD Ryzen 9 7945HX (16 cores, 32 threads), 64 GB RAM, one consumer NVMe (Corsair MP600 Elite: volatile write cache, no power-loss protection), ext4, Linux 6.8. Everything runs on this one host, client included.
Systems:
| Version | Layout | Durability of an acknowledged PUT | |
|---|---|---|---|
| IOLake | main of 2026-10-04, release build (isal, io-uring) |
iolake-aio: meta, six OSDs and the gateway in one process; 4+2 erasure coding; each OSD a 64 GiB sparse file |
shards at write quorum (5 of 6), each fsynced |
| RustFS | rustfs/rustfs:latest (image of 2026-10-03) |
one server, six directories, EC:2 (4+2) |
erasure-coded at write quorum, fsynced |
| SeaweedFS | 4.48 | weed server (master, volume, filer, S3) with its defaults |
one copy, not fsynced per write |
The six "drives" of IOLake and RustFS share one device here. That is fine for comparing the software, not for absolute numbers: on separate drives both would write in parallel.
SeaweedFS's defaults keep one copy and don't fsync each write, so its PUT numbers are not comparable with the other two. They show what that trade-off buys, not a like-for-like result.
Client: warp 1.1.4. For each object size, PUT then GET, 32
concurrent clients, 30 s per cell. The script is
scripts/bench/baseline.sh in the code repo.
Results¶
Throughput (p99 latency). IOLake and RustFS ran each cell up to 3 times; the range is shown where the runs differed by more than 10%.
| Cell | IOLake | RustFS | SeaweedFS (one copy, no fsync) |
|---|---|---|---|
| PUT 4 KiB | 553–558 obj/s (195 ms) | 353–477 obj/s (135 ms) | 21,388 obj/s (4.3 ms) |
| PUT 64 KiB | 236–239 obj/s (223 ms) | 348–373 obj/s (145 ms) | 13,913 obj/s (7.5 ms) |
| PUT 1 MiB | 111–241 MiB/s (423 ms) | 183–217 MiB/s (239 ms) | 1,595 MiB/s (168 ms) |
| PUT 16 MiB | 486–772 MiB/s (1.3 s) | 503 MiB/s (2.8 s) | failed (internal errors) |
| GET 4 KiB | 8,403–10,511 obj/s (5.9 ms) | 31,692–32,918 obj/s (2.0 ms) | 20,203 obj/s (3.9 ms) |
| GET 64 KiB | 5,392–5,808 obj/s (9.7 ms) | 25,613–27,141 obj/s (2.4 ms) | 18,931 obj/s (4.2 ms) |
| GET 1 MiB | 1,785–1,954 MiB/s (25.7 ms) | 2,419–2,479 MiB/s (105 ms) | 3,734 MiB/s (16.7 ms) |
| GET 16 MiB | 1,716 MiB/s (415 ms) | 2,304 MiB/s (386 ms) | not run (the PUT failed) |
What it shows¶
- Durable PUTs are bound by the disk. IOLake and RustFS both fsync before they acknowledge, and both land in the hundreds of small PUTs a second on this drive. A drive with power-loss protection makes an fsync nearly free; this one doesn't. SeaweedFS is 40–60 times faster because it skips that work.
- 64 KiB PUTs: RustFS is 1.5 times faster. RustFS writes each drive's
erasure-coded piece of a small object inside that drive's metadata file
(
xl.meta; seen here for 64 KiB and 256 KiB objects), so a small PUT is one fsync round. IOLake inlines only objects up to 4 KiB (--inline-max-size), whole, on every copy. A 64 KiB object takes two rounds: shards, then metadata (the small-object path, since, makes it one). - Small GETs: RustFS is 3–4.5 times faster. RustFS reads its own drives. IOLake's aio is a cluster in one process: a GET makes one RPC to meta for the placement and asks all six OSDs for the object's metadata, then reads shards (none for an inline object). That is 7 RPCs for a 4 KiB GET and 11 for 64 KiB, each over loopback HTTP/2. Under profiling, nearly all the CPU goes to that RPC work and to lock contention inside h2. In a multi-host deployment RustFS makes these calls too, so this gap is partly an artefact of one host. But fewer round trips per small GET is a real gain everywhere (since built: the small-object path).
- Large objects: within 1.3 times. At 1 MiB and above both are close to the device and to the CPU cost of erasure coding.
Found along the way and fixed:
- warp could not create buckets: the gateway refused
PUT /bucket/with the trailing slash that minio-go clients send. - The OSD's metadata cache scanned its lists on every lookup: +7% on small GETs.