Skip to content

How it works

IOLake is three kinds of service around one storage core. This page follows an object through it: where it goes when it's written, how it's found and checked when it's read, and what happens when a disk or a host fails.

The services

           S3 / Iceberg / Unity Catalog / Delta Sharing / admin
                                │
                     ┌──────────┴──────────┐
                     │  Gateways (9000)    │  stateless: as many as you like
                     │  auth, erasure code │
                     └───┬─────────────┬───┘
            placement,   │             │   shards and object
            listings,    │             │   metadata, in parallel
            IAM          │             │
          ┌──────────────┴───┐   ┌─────┴──────────────────────────┐
          │ Meta (9100)      │   │ OSDs (9200), one per host      │
          │ Raft, 3+ nodes   │◀──│ raw disks: shards, and each    │
          │ one redb file    │   │ object's metadata copy         │
          │ each             │   │                                │
          └──────────────────┘   └────────────────────────────────┘
Service Holds Scales by
Gateway Nothing: every request is served from meta and the OSDs. It speaks S3 (SigV4), the data-lake APIs and the admin API, and does the erasure coding Adding gateways behind a load balancer
Meta Everything shared that isn't object bytes: buckets, IAM users and policies, tenants, pools and placement groups, bucket listings, catalog tables. Every write goes through Raft; any node answers, followers forward writes to the leader 3 or 5 nodes; a majority keeps it writable
OSD Shards on raw block devices it owns outright, and a copy of the metadata of every object it holds a shard of One per host (or per disk set); add hosts

iolake-aio runs all of them in one process, for a laptop or an appliance; the wire protocol is the same. Between services, every gRPC hop can be mutual TLS.

Placement

An object's key hashes (jump consistent hash) to one of a pool's placement groups. Each placement group is a fixed list of OSDs, one per shard, chosen so that no two share a failure domain: a host by default, or a rack, zone or region if you declare them. Meta records that list; the balancer moves whole placement groups, never single objects, so rebalancing costs the number of placement groups, not the number of objects.

A pool is an erasure code (4+2 or 8+4 Reed-Solomon, or replicas) over a set of OSDs. A cluster grows by adding OSDs to a pool or by adding a pool, never by splitting placement groups.

Writing an object

With 4+2 erasure coding:

  1. The gateway authenticates the request and asks meta for the key's placement group.
  2. It splits the object into 4 data shards and computes 2 parity shards (with ISA-L, on x86_64 and arm64 alike), with a CRC32C of each.
  3. It writes the six shards to the six OSDs in parallel. Each OSD writes its shard to raw disk with O_DIRECT, flushes the drive's cache, then records and flushes the shard's location. The write needs 5 of the 6.
  4. It writes the object's metadata (size, ETag, version, stamp, where each shard is and its checksum) to every OSD of the placement. This needs a majority: 4 of 6.
  5. It records the key in the bucket's listing, in meta, through Raft.
  6. Only then does the client get its 200.

Objects up to 4 KiB skip the shards: they are kept whole inside their metadata, on every copy. Small objects written as separate stripes are later packed together in the background, so a million 8 KiB objects don't each take a stripe of their own.

A multipart upload writes each part this way and records the parts in one object at completion.

Reading an object

  1. The gateway reads the object's metadata from the OSDs of its placement and takes the copy with the highest stamp among a read quorum (3 of 6). Any 3 copies include one that took the last acknowledged write, so a copy that missed an overwrite or a delete is outvoted, never served.
  2. It reads 4 shards, preferring the nearest OSDs, and checks each against the checksum recorded in the metadata.
  3. If a shard is missing or doesn't match, it reads another and reconstructs: any 4 of the 6 rebuild the object. Each data shard it rebuilds is checked against its recorded checksum before it is served; one that doesn't match is rebuilt from other shards, or the read fails. The bad shard is reported for repair.

So with 4+2, any two OSDs can be down and every object still reads and every write still succeeds.

Inside an OSD

  • Raw disks. No filesystem: a superblock (kept twice), an allocation bitmap of 4 KiB blocks, and the shards. Each shard carries a header and footer with its own checksum. I/O is O_DIRECT, through io_uring on Linux.
  • Its metadata store. A write-ahead log, flushed before every acknowledgement, in front of an on-disk index with a bounded cache. The OSD's memory stays flat however many objects it holds.
  • Scrub. In the background, at a set rate, the OSD reads every shard it holds and checks it, so a shard that rots on disk is found even if nobody reads it.
  • It never formats a disk that holds data. A disk is formatted only if it's blank: a disk the OSD can't recognise is reported, not wiped.

When something fails

Fault What happens
A disk or an OSD fails Reads reconstruct from the remaining shards; writes go on at quorum. Repair rebuilds the missing shards from the survivors, checks each against its recorded checksum, and places it on another OSD in the same placement rules
A host loses power The same. Every acknowledged write was flushed to stable storage, so it's there when the host returns; copies it missed are healed
A shard rots on disk Scrub or a read finds the checksum mismatch; the shard is rebuilt from the others
The meta leader fails The others elect a new leader in a few seconds; gateways retry. Writes resume, nothing acknowledged is lost
Meta loses its majority Writes are refused with 503 (retry), never accepted and lost; they resume when the majority is back
A network partition The side with the meta majority carries on; the cut-off side refuses writes. When the partition heals, the cut-off nodes catch up
A gateway's clock drifts Writes are stamped with the gateway's clock and the newest stamp wins, so a gateway more than 500 ms off meta's clock refuses writes until it's back in step
A disk fills Client writes stop at the full ratio (507); the space left is kept for repair
You remove a disk or host Drain moves its shards off first, then it can go

Each of these is exercised by the chaos tests or the end-to-end suite on every pull request.

Data lake and block on the same core

  • The Iceberg REST Catalog and the Unity Catalog API keep their tables in meta and their data in buckets, so one deployment is the catalog and the storage. A Delta Sharing server shares tables from any tenant.
  • Block volumes (NBD) are 4 MiB chunks erasure-coded the same way, with a write journal flushed before each acknowledgement, and snapshots and clones that share stripes.

Features has the status of each, and the roadmap what's next.