¶
S3-compatible object storage in Rust, built to not lose data. One storage core (erasure coding, failure-domain-aware placement, repair, metadata on Raft) with S3, data-lake catalogs and block volumes on top of it. Apache-2.0, every feature, no tiers. Source on GitHub.
docker run --rm -p 9000:9000 -v iolake-data:/var/lib/objectio \
ghcr.io/io-lake/iolake:latest iolake-aio --data /var/lib/objectio
That's a whole cluster in one process: meta, OSDs and the S3 gateway on port 9000. The quickstart has the binaries for Linux and macOS, and a 3-meta, 6-OSD cluster in Docker Compose; Kubernetes has the Helm chart.
Where it stands¶
IOLake 0.1.0 is the first release, and an early one (release notes). S3 and the storage core are tested for production; the data lake APIs and block volumes are previews. Every feature and how far it has been proven is in features; what's next is in the roadmap.
| Status | |
|---|---|
| S3 | Passes 680 of 687 applicable ceph s3-tests; the other 7 are written down, none a bug. Versioning, Object Lock, lifecycle, multipart, checksums, SSE, bucket policies, IAM and STS APIs, tenants, replication between clusters, S3 Select, audit |
| Storage core | Erasure coding (Reed-Solomon with ISA-L, on x86_64 and arm64), placement groups across failure domains, recovery and repair by placement group, drain, backfill, balancer, small-object packing, scrub; raw disks with O_DIRECT and io_uring |
| Meta | Raft (openraft) on 3 or more nodes; any node serves; its own log, back within a second of a power cut |
| Data lake (preview) | Iceberg REST Catalog, Unity Catalog API and Delta Sharing are built; tests with real engines (Spark, Trino, pyiceberg) come next |
| Block (preview) | Volumes over NBD: thin provisioning, snapshots, clones, a write journal fsynced before the ack. Filesystem and database crash tests are the gate for production |
| File | NFS on the same core, later |
What "durable" means here¶
- An acknowledgement means the data is on stable storage. Every shard is fsynced, and its location record after it, before the PUT returns. Nothing trades this for speed.
- No single copy decides. An object's shards are spread across failure domains (hosts by default; racks or zones if you declare them), and its metadata is kept with them on every OSD of its placement, written at a majority and read from a quorum, newest copy winning.
- Every byte is checked twice. Each shard's checksum is recorded with the object when it's written and checked on every read and every repair, independently of the checksum the disk keeps.
- Repair is continuous. A lost disk or host is rebuilt from the surviving shards onto the rest of the cluster, without an operator.
How it works has the detail.
How it's tested¶
| Test | What it shows | Latest (0.1.0 release candidate) |
|---|---|---|
| S3 compatibility | ceph/s3-tests, the suite Ceph RGW is built against; every failure labelled with why | 680 of 687 applicable pass |
| Chaos | A six-machine cluster under writes from two gateways while the meta leader is killed, hosts lose power, the network is cut, disks are pulled and drives lost for good. Every acknowledged object is read back after each fault | Passes; 81,436 writes intact, writes paused at most 13.2 s |
| Crash consistency | The OSD's disk writes recorded and replayed up to random points, as a power cut would leave them: every acknowledged write must survive | 80 cuts, nothing lost |
| Long soak | 48 hours on a lab host: overwrites and deletes, disks filling, power cuts, partitions, drives lost, repair, and an audit of every key at the end | Passes: 84 faults, 7.02 M operations, nothing lost or leaked |
| Performance | Side by side with RustFS and SeaweedFS, same client, same disk | Single host and six cloud VMs published |
| Every change | Unit and end-to-end tests against real processes, a meta HA test on Kubernetes, a rolling upgrade from the last release under traffic, and a performance check |
Read next¶
- How it works: the services, the write and read paths, placement, repair.
- Features: every feature, by domain, and how far it has been proven; the roadmap has what's next, by release.
- Operations guide: install, administer and run a cluster; the production guide, and hardware design and sizing with a calculator from requirements to a bill of quantities.
- Developer guide: the S3, Iceberg, Unity Catalog, Delta Sharing and admin APIs; the feature reference.
These pages follow the main branch (dev); the pages for each release are in the version selector at the top.