Hardware design and sizing¶
How to go from a requirement (so many terabytes, so many GB/s) to a bill of quantities: how many hosts, which drives, how much CPU, memory and network, and where meta and the gateways run. The calculator at the end does the arithmetic; the sections before it say where each number comes from.
IOLake is an early release (0.x) and its multi-host performance numbers are still being measured. Each figure here says whether it was measured (and where) or is an estimate. Size from the estimates with headroom, and check them on your own hardware before you buy at scale.
Where to start¶
Work through these in order; each one can only raise the count the previous one gave.
- Protection. Choose the erasure code: 4+2 for 7 to 12 hosts, 8+4 from 13. It sets the overhead (1.5× raw for both) and the minimum number of hosts: one per shard plus one to rebuild onto (7 for 4+2, 13 for 8+4).
- Capacity. Usable terabytes, with the headroom you want, gives raw terabytes, then drives and hosts.
- Throughput. GB/s of writes and of reads gives the drives and hosts the bytes need, and the network every host needs.
- Objects. The average object size gives the object count, which sizes the metadata: meta's disks and each drive's metadata partition.
- CPU and memory. From drives per host, and from the GB/s the gateways carry.
- Placement of services. Meta and the gateways on the storage hosts (converged), or on their own servers.
The rules behind the numbers¶
Protection and hosts¶
| Code | Raw per usable byte | Survives | Hosts at least |
|---|---|---|---|
| 4+2 | 1.5 | any 2 hosts | 7 |
| 8+4 | 1.5 | any 4 hosts | 13 |
| 10+4 | 1.4 | any 4 hosts | 15 |
Placement puts each shard of a stripe on a different host. With fewer hosts than shards it would put two on one host, and losing that host would lose both: the minimum is one host per shard, plus one so a lost host can be rebuilt onto the others.
Capacity¶
usable = raw × k/(k+m) × 0.95 − one host's worth
- 0.95: an OSD refuses client writes past 95% of its drive
(
--full-ratio) and keeps the rest for repair, drain and backfill. - One host's worth: when a host is lost for good, its shards are rebuilt onto the others; keep that much free.
- Add your own headroom on top: growth, and the "near full" alert at 85%.
Objects of 4 KiB or less are kept whole inside their metadata, on all
k + m copies (--inline-max-size): they take metadata space (below), not
data space. Small objects in general are packed into shared stripes in the
background, so a million 8 KiB objects don't each take a stripe.
Drives¶
- Power-loss protection (PLP). An acknowledgement means the data is on stable storage: every shard and its location are flushed before the client hears back. On a drive without PLP a flush is slow (it empties a volatile cache) and a power cut can still lose what the cache held. Use enterprise NVMe with PLP; otherwise a UPS for every host.
- One OSD per drive. The shards go on the raw device; the OSD's metadata (its write-ahead log and index) goes on a small partition of the same drive, not the boot disk (see Objects and metadata for its size).
- Boot: two small drives (M.2, mirrored) for the OS.
Throughput¶
What a write costs, with a k+m code:
| Per GB/s of client writes | Per GB/s of client reads | |
|---|---|---|
| Bytes written to drives | (k+m)/k GB/s (1.5 for 4+2) | — |
| Bytes read from drives | — | 1 GB/s (k shards of the object) |
| Gateway → OSD network | (k+m)/k GB/s | 1 GB/s (OSD → gateway) |
| Client ↔ gateway network | 1 GB/s | 1 GB/s |
How much one drive sustains under IOLake is not yet measured on enterprise NVMe. The calculator assumes 1 GB/s of writes and 2 GB/s of reads per drive, a fraction of what a PCIe 4 drive is rated for; change it to what you measure.
Small objects are bound by operations, not bytes. Measured on one lab host (consumer NVMe without PLP, everything on one machine, 4+2, 32 clients): 1,150–1,300 PUTs/s and 25,900–27,000 GETs/s of 4 KiB objects, 990–1,100 PUTs/s and 17,600–18,200 GETs/s of 64 KiB; the performance page has the details. A drive with PLP makes each flush much cheaper. Treat small- object rates as something to measure on your hardware.
CPU¶
Measured per core on an AMD Ryzen 9 7945HX (Zen 4), 4 MiB stripes:
| Work | GB/s per core |
|---|---|
| Erasure coding, ISA-L, 4+2: encode / decode with 2 shards lost | 9.9 / 11.3 |
| Erasure coding, ISA-L, 8+4 | 8.8 / 9.3 |
| MD5 (every PUT's ETag) | 0.80 |
| SHA-256 (clients that sign the payload; CPU with SHA extensions) | 2.4 |
| CRC32C (every shard, written and read) | 8.4 |
On arm64 erasure coding is ISA-L's too. One Apple M-series core, in a Linux VM, 4 MiB shards: 4+2 encodes at 23.8 GB/s and decodes with 2 shards lost at 15.6 GB/s; 8+3 at 16.2 and 4.4. A server arm64 core (Graviton, Ampere) is not yet measured.
What a gateway spends per GB/s:
| Writes | Reads | |
|---|---|---|
| MD5 for the ETag | 1.25 cores | — |
| SHA-256, if clients sign the payload (most SDKs over HTTPS don't) | 0.4 | — |
| Erasure coding and checksums | 0.3 | 0.15 |
| Network stack and HTTP (estimate) | 1–2 without offload; about 0.3 with TSO/GRO offload or RDMA | 1 without; about 0.2 with |
| Total | about 3 without offload, 1.6 with | about 1.2 without, 0.35 with |
The MD5 is computed in the gateway over every byte written, and no NIC can take it: it sets the floor. Offloads take the network stack, and the shard traffic can go over RDMA (behind a flag today). mTLS between services costs about 0.35 cores per GB/s of shard traffic (AES-GCM, estimate).
An OSD's own work is light (checksums, I/O submission, RPC): plan about one core per NVMe drive (estimate), plus a few for the OS.
Memory¶
| Plan | |
|---|---|
| OSD | 1.5 GB per drive: its metadata index cache (--meta-cache-mib, 1 GiB by default), memtables, buffers. Flat however many objects it holds |
| Gateway | 4 GB, plus 2 GB per GB/s it carries (buffers for objects in flight). Set it as the gateway's memory limit: half of it becomes its memory budget, and load beyond that gets 503 SlowDown |
| Meta | 16 GB; 32 GB above 500 million objects (its database cache) |
| OS | 8 GB per host |
Measured on the soak cluster (6 hosts, 69,000 objects, before the caches filled): OSD 164 MB, gateway 100 MB, meta 91 MB resident.
Objects and metadata¶
The object count (usable bytes ÷ average object size) sizes the metadata. Measured on the soak cluster:
| Where | Per object |
|---|---|
| Meta's database, on every meta node | about 750 bytes (its listing entry and the rest) |
| An OSD's metadata index, on every OSD holding a copy (k + m of them) | about 1.5 KB (the object's metadata copy and its shard's location) |
Meta also keeps its Raft log and one snapshot of the database beside it,
a second while the next snapshot is built, and a reserve free
(--min-free-mib, 1 GiB) under which it takes no writes: plan 3.2 × the
database, plus 1 GiB, on each meta node's disk
(meta's disk filling).
Each drive's metadata partition must hold its share of copies:
index per drive ≈ objects × (k+m) ÷ drives × 1.5 KB
An object whose shards are 16 KiB or less (64 KiB or less with 4+2) keeps them in the index too: add the object's size ÷ k to each copy's 1.5 KB (from format level 8; before it, a shard just past a power of two, such as a 64 KiB object's 16 KiB with 4+2, took twice its size there). At a 1 MiB average object the index is under 1% of the drive; at 64 KiB, where the shards live there, the whole drive's worth of data is in the partition. The calculator gives the partition size.
Network¶
- Two ports per host, bonded or one per switch, to two top-of-rack switches: losing a switch or a cable must not take a host away.
- Size for repair too. Rebuilding a lost host moves its whole contents across the network; the calculator doubles the client traffic to leave room for it.
- Client-facing TLS terminates at the load balancer (or an ingress); between services, mTLS.
Drive failures, replacement and hot spares¶
A failed drive's data is rebuilt at once, onto the free space of every other drive in the cluster: many drives write in parallel, so it takes minutes to hours where rebuilding onto one spare disk, as a RAID array does, is held to one drive's write rate. Replacing the drive afterwards only gives its capacity back. So how fast a vendor replaces drives decides how long capacity is down, not how long data is at risk, as long as there is free space to rebuild into.
- Spare space, not idle disks. The calculator holds one host's worth of drives free for rebuilds. With the drives' failure rate and the time to replace one (on-site spares, next business day, an OEM's five days), it works out how many drives may be failed and not yet replaced at once (99.9%) and says when that is more than the space held: add hosts or drives, keep hot spares, or shorten the replacement time.
- Hot spares (drives installed but not in use) make sense where replacement is slow: a remote site, an OEM contract measured in days. One is put in service in minutes, with no site visit, and the capacity is back; restock them at least as fast as drives fail. They don't make data safer: the rebuild has already happened on the other drives.
- Rebuild times in the calculator are what the drives and the network allow with 30% of them given to repair: a bound. Today's repair is slower than that; raising it is on the roadmap.
Where meta and the gateways run¶
- Meta: three nodes (five to survive two at once), each on its own host, on NVMe with PLP. Up to a few hundred million objects they can share three of the storage hosts; past that, give them servers of their own.
- Meta gets a drive of its own (not an OSD's drive, not the boot disk), even on a shared host. Every PUT waits for meta's commit, and an OSD's metadata checkpoint writes its index in bursts that hold back other syncs on the same drive. Measured with meta and six OSDs on one NVMe: meta's checkpoints took 0.25–1 s whenever an OSD checkpointed, and every PUT in flight waited (64 KiB PUTs: p99 up to 580 ms against a 38 ms median). With meta on its own drive, none took 0.1 s. A small drive does: meta needs 2.2 × its database (above).
- Gateways: stateless, behind an L4 load balancer. Converged (one on every storage host) spreads the CPU and network evenly and needs no extra servers; dedicated gateway servers make sense when the storage hosts are small or the GB/s are high.
Calculator¶
Fill in the requirement; the bill of quantities updates as you type. It runs in your browser and sends nothing anywhere.
What isn't in the calculator yet¶
- Repair time as measured. The calculator gives what the hardware allows; repair's own throughput is being raised to meet it (on the roadmap).
- Small-object operation rates. See Throughput: measure them on your hardware.
- Multi-site. Replication between clusters doubles the hardware, one cluster per site, plus the link between them.