IOLake features¶
Everything IOLake does today, by domain, and how far each part has been proven. A feature on this page exists; one that isn't here doesn't (what is planned is in the roadmap). How each feature works, with its wire format and settings, is in the feature reference and the guides.
Reading the columns
- Status: ✅ shipped and covered by end-to-end tests; 🟡 shipped, but not yet proven for production (the gap is named); ✖ not supported, by decision (the reason is given).
- Guide: where a user or operator reads how to use it.
- Design: how it works inside.
How far 0.1.0 has been proven¶
0.1.0 is the first release, and an early one. S3 and the storage core are tested for production; the data lake APIs, block volumes and file access are previews.
- S3: ceph s3-tests, 680 of 687 applicable tests pass at the release candidate on Linux x86_64, 679 on Linux arm64; the rest are written down (6 known gaps, and tests caught by another running beside them, which pass alone). None is a bug.
- The storage core: a 48-hour soak of 84 faults (power cuts, meta leader kills, network partitions, disks pulled and drives lost for good) under 7.02 M acknowledged operations and 1.50 TB written: no acknowledged object lost, no space leaked. The chaos suite on six machines: 81,436 acknowledged writes intact through every fault. Recorded disk writes replayed up to random power-cut points: nothing acknowledged lost.
- Data lake, block, file: the data lake APIs and block volumes are built and unit- or end-to-end-tested, but not yet tested with real engines (Spark, Trino, pyiceberg) or under a filesystem and a database; file access is not built.
Adjacent releases upgrade one node at a time, under traffic, with every object readable after (upgrades). The admin API and the CLI may still change between 0.x releases; the release notes say when they do.
S3¶
| Feature | Status | Notes | Guide | Design |
|---|---|---|---|---|
| Buckets: create, delete, head, list, location | ✅ | GetBucketLocation answers the gateway's --region (us-east-1 as the empty constraint, as S3 does) |
S3 API | |
| Objects: PUT, GET, HEAD, DELETE, DeleteObjects | ✅ | S3 API | ||
Range reads, conditional requests (If-*), conditional writes (If-None-Match: *, If-Match) |
✅ | S3 API | ||
| Multipart upload (every call, 10,000 parts) | ✅ | Completion is two-phase in meta. Parts up to 5 GiB, objects up to 5 TiB, as S3 | S3 API | |
| A single PUT and an upload part up to S3's 5 GiB | ✅ | A body over 16 MiB is streamed to the shards stripe by stripe, never held whole: the gateway holds a few stripes of it whatever its size. Committed only once all of it has arrived and passed its payload hash, chunk signatures and checksums; a larger body gets 400 EntityTooLarge before it is read. A gateway that stops mid-upload leaves its shards until the orphan scan (roadmap) |
S3 API | |
| Addressing: path-style, and virtual-hosted under the gateway's S3 domains | ✅ | Virtual-hosted needs --s3-domain, wildcard DNS and a wildcard certificate. A request whose bucket can't be told for certain is refused, not misread; without a domain, a bare GET / is still ListBuckets |
S3 API | |
| CopyObject, UploadPartCopy, copy by reference | ✅ | A copy that changes the encryption rewrites the bytes, streamed as a large PUT is, from a source of up to 5 GiB | S3 API | |
| Gateway memory bounded under any load | ✅ | A GET streams stripe by stripe, and so does a large PUT or upload part; request bodies and read-ahead come from a memory budget; requests over the budget or the in-flight limit get 503 SlowDown; slow clients are cut off by timeouts |
production | |
x-amz-meta-*, content headers, response-* overrides |
✅ | S3 API | ||
| Checksums (CRC32, CRC32C, SHA1, SHA256; aws-chunked) | ✅ | S3 API | ||
| Object tagging | ✅ | S3 API | ||
| Versioning, delete markers, ListObjectVersions | ✅ | feature reference | ||
| Object Lock (governance, compliance, legal hold) | ✅ | feature reference | ||
| Lifecycle: expiration, noncurrent versions, delete markers, abort incomplete uploads | ✅ | feature reference | ||
| Lifecycle transitions | ✖ | One storage class; refused with InvalidStorageClass. Revisited with tiering |
||
| CORS, presigned URLs, POST Object | ✅ | feature reference | ||
| S3 Select | ✅ | CSV, JSON (document, lines) and Parquet input, GZIP and BZIP2, CSV or JSON output, scan ranges; AWS's SQL subset. 29 of 34 ceph s3-tests pass; the other 5 expect RGW's dialect | S3 API | S3 Select |
Gateway-side grep (?grep on an object or a prefix) |
✅ | Matches streamed as NDJSON with byte offsets; regex, pcre2, hyperscan |
grep | |
Bucket logging (?logging, S3 access-log format) |
✅ | Delivered through the audit pipeline into the tenant's target bucket | bucket logging | bucket logging |
| Audit stream (webhook, file, stdout; per tenant) | ✅ | One event per request, with the request ID the client got | audit | |
| Audit that loses nothing | ✅ | Events fsynced to a spool on the gateway before delivery, a cursor per target: a gateway killed and a receiver down for an hour lose no event | audit | |
| Audit on by default for the system | ✅ | Every event into an internal bucket (the Helm default), with retention | audit | |
| Bucket replication, one way, between IOLake clusters | ✅ | New writes as they happen; targets scoped to a tenant | replication | bucket replication |
| Replication resync of existing objects, bandwidth limit | ✅ | A resync resumes from its last page after a kill; a 256 KiB/s limit held over every 4 s window | replication | bucket replication |
| Quotas: bucket count per tenant | ✅ | Enforced by meta | IAM | |
| Quotas: bytes and objects, per tenant and per bucket | ✅ | 403 QuotaExceeded; overshoot across gateways is bounded |
IAM | quotas |
| Static website hosting, inventory, analytics, transfer acceleration, torrent | ✖ | Refused with 501, never misread; so is a bucket GET naming any sub-resource the gateway doesn't answer | S3 API |
Core¶
| Feature | Status | Notes | Guide | Design |
|---|---|---|---|---|
Raw-disk OSD: O_DIRECT, 4 KiB blocks, a write-ahead log |
✅ | deployment | storage engine | |
| Flush before acknowledge | ✅ | Each shard fsynced, then its location record; a drive's volatile cache flushed on every write; a failed location record fails the write | production | storage engine |
| One write quorum for every stripe | ✅ | PUTs, multipart parts, packs and block chunks alike: k + 1 shards (MDS), two copies, or all but g − 1 (LRC), counted on distinct OSDs; 503 short of it (507 when full OSDs kept it short). Every shard carries its placement group's epoch, and a write placed under an old one is placed again | S3 API | data protection |
| Crash consistency of the OSD | ✅ | The shard device's and state filesystem's writes recorded and replayed up to random power-cut points: every acknowledged write survives. Run in CI on every merge | storage engine | |
| On-disk format checks | ✅ | Superblock feature words, protobuf records, strict logs and snapshots | disk layout | |
| Superblock safety | ✅ | Two copies; a torn one mounts from the other; deletes no longer rewrite it | disk layout | |
| An OSD's disks known by their ids | ✅ | Each disk's index recorded by the id in its superblock: disks found in any order or at other paths; an OSD missing one, or given one not its own, doesn't start; a blank disk replaces a missing one. Reads and scrub check each block is the shard its location names (object, stripe, length, CRC32C) | production | storage engine |
| OSD metadata store (WAL, on-disk index, bounded cache) | ✅ | Memory flat with the object count; snapshots streamed, not cloned | storage engine | |
| OSD back in seconds after a restart | ✅ | An OSD of 10 M shards serves 16.9 s after a power cut; its counts are rebuilt after it serves | failure recovery | OSD start |
| io_uring shard I/O | 🟡 | The default in Linux release builds; a host that denies io_uring (docker's default seccomp profile) gets pread, with a warning. The data-path end-to-end tests run on it in CI (S3 calls, multipart, repair, restarts, packing, checksums, removed disks); the rest of the suite and the soak on pread |
Docker | I/O levels |
| Erasure coding: Reed-Solomon with ISA-L, on x86_64 and arm64 | ✅ | 4+2 by default. One code on every build, its parity checked against the reference code on both architectures, so amd64 and arm64 nodes mix in one cluster (not yet run as one). A pool takes only profiles with parity that every loss of m shards decodes | Linux | data protection |
| Reads never serve a data shard rebuilt wrong | ✅ | Each data shard a read rebuilds is checked against its recorded checksum (or a spare parity shard) before it is served; a decode that doesn't check out is tried another way, then fails and reports the stripe for repair | data protection | |
| LRC | 🟡 | Objects of any length, PUT or multipart, read back with a drive lost; repair built and tested. Survives every loss of its global parity count (LRC 6+2+2: any 2, 88% of 3); not yet through the full end-to-end suite or a soak | CLI | data protection |
| Replica pools (2-way, 3-way) | 🟡 | Repair built and tested; not yet through the full end-to-end suite or a soak | data protection | |
| End-to-end shard checksums, scrub | ✅ | Each shard's checksum kept with the object and checked on every read and repair, besides the OSD's own | data protection | |
| Object metadata at quorum | ✅ | With one or two OSDs down (4+2), PUT, overwrite and DELETE go on; reads get the newest copy, never a deleted object | object metadata quorum | |
| Small-object path | ✅ | A small object's shards travel with its metadata, one round trip; a small GET reads metadata from a read quorum | performance | small-object path |
| Small-object packing (pack, reconcile, compact) | ✅ | packing | ||
| Placement: CRUSH2, placement groups, copyset allocator | ✅ | feature reference | topology | |
| Failure-domain topology (5 levels), locality-aware reads | ✅ | topologies | topology | |
| Refusing placements the topology can't satisfy | 🟡 | A validation endpoint; enforcement on write is planned | topology | |
| Storage pools (expand by adding a pool), balancer | ✅ | A pool's protection scheme is checked when it is made or changed: one the codecs can't keep (no parity, LRC groups of unequal size, no copies) is refused | feature reference | data protection |
| Recovery by placement group | ✅ | Every pool has PGs whose acting set and epoch are committed before a write uses them; OSDs refuse writes from an older epoch; peering and a state machine per PG | failure recovery | PG recovery |
| Repair after a disk or host is lost | ✅ | Rebuilt from the surviving shards onto the rest of the cluster, without an operator; parallel across PGs, sized from the drives | failure recovery | PG recovery |
| Degraded objects recorded when written, repaired first | ✅ | A PUT acknowledged short of shards is recorded in the Raft commit that creates it | degraded objects | |
| An OSD lost for good is re-protected | ✅ | A drive gone with its metadata, set out: every shard it held is rebuilt elsewhere | failure recovery | PG recovery |
| Backfill | ✅ | A stripe with two shards on one OSD is re-spread once a distinct OSD can take one | PG recovery | |
| Drain (planned removal of an OSD) | ✅ | failure recovery | ||
| Full-disk behaviour | ✅ | Client writes stop at the full ratio (507 StorageFull); the rest is kept for repair; space returns after deletes. A full filesystem under an OSD (ENOSPC, EDQUOT) is 507 too, and writes resume once there is room |
production | |
| Meta: Raft (openraft), replicated state, snapshots, log compaction | ✅ | A node that was down longer than the kept log catches up from a snapshot | meta HA | meta |
| Meta's own Raft log | ✅ | Sealed segments open from an index; a meta node listens within a second of a power cut, a torn last append cut off | meta log | |
| Bucket listings that cost the same per page anywhere in the bucket | ✅ | The scan seeks to the continuation marker and past each common prefix: paging through 100,000 keys reads each row once | meta | |
| Meta HA: any node serves, failover, node loss | ✅ | Leader killed, frozen and replaced under S3 traffic, in multi-process tests and on Kubernetes (3 metas through Helm) | meta HA | meta |
| Meta reads confirmed by a quorum (read index) | ✅ | The leader answers only while a majority has confirmed it leads within the last second; one cut off from its followers answers UNAVAILABLE within about 3 s, in a multi-process test |
meta HA | meta |
| Rolling upgrade, format levels | ✅ | Meta, then OSDs, then gateways, under traffic; new formats gated until every node runs the new release; tested on every change to main |
upgrades | upgrade path |
| RDMA data plane (Mooncake Transfer Engine) | 🟡 | Single-part erasure-coded objects, behind a flag | RDMA data plane |
Data lake¶
Previews: built and unit-tested, not yet tested with real engines (the roadmap has the engine tests as the next release's first items).
| Feature | Status | Notes | Guide | Design |
|---|---|---|---|---|
| Iceberg REST Catalog (namespaces, tables, commits) | 🟡 | Unit tests only; no engine tests | Iceberg API | |
| Warehouses (a backing bucket each) | 🟡 | Iceberg API | ||
| Namespace and table policies, data filters | 🟡 | Iceberg API | ||
| Vended credentials (STS) | 🟡 | Iceberg API | ||
| Unity Catalog API (catalogs to model versions; row filters, column masks) | 🟡 | Unit tests only | feature reference | |
| Delta Sharing (shares, recipients, presigned query) | 🟡 | Unit tests only; Delta tables only (an Iceberg table is refused); a share exposes its own tenant's data, and only the system admin adds a table of another tenant | feature reference | |
| Delta log: JSON commits | 🟡 | Checkpoints (Parquet) not read yet | feature reference |
Security¶
| Feature | Status | Notes | Guide | Design |
|---|---|---|---|---|
| Users, access keys, groups, policies | ✅ | IAM | ||
IAM API (aws iam, boto3, Terraform) |
✅ | Users, groups, managed and inline policies, keys and roles, scoped to a tenant; the ceph s3-tests IAM tests pass or fail only for documented reasons | IAM | |
| STS: AssumeRole, AssumeRoleWithWebIdentity, GetSessionToken, GetCallerIdentity | ✅ | Identity providers per scope: the operator's for system roles, each tenant's for its own | IAM | |
| Tenants, tenant admins, tenant isolation | ✅ | A tenant is an account, its admins its root; disabling one refuses all its identities within 15 s | IAM | |
| OIDC sign-in (external providers: Keycloak, Entra ID, …) | ✅ | Console sign-in is the user bound to the token's issuer and subject; a tenant's provider signs in to that tenant only (IAM). A tenant names only a stored provider, and a provider a tenant names can't be deleted | authentication | |
Bucket policies (conditions, obio:CredentialType) |
✅ | authentication | ||
| Anonymous access and Block Public Access (bucket, tenant, cluster) | ✅ | New buckets start blocked | public access | |
Named endpoints and client addresses in policies (aws:SourceVpce, aws:SourceIp, trusted proxies) |
✅ | public access | ||
| ACLs | ✖ | Bucket owner enforced only, AWS's default; use policies | ||
| SigV4 (header, query, streaming) | ✅ | The body is held to its signed hash; each streaming chunk, and a signed trailer, to its signature | authentication | |
| SigV2 | ✖ | Deprecated by AWS | ||
| SSE-S3, SSE-KMS (built-in KMS) | ✅ | feature reference | ||
| SSE-C | ✅ | Not yet on a CopyObject source or destination | feature reference | |
| Tenant-scoped audit and replication targets | ✅ | Host allowlist set by the operator | audit | |
| Admin and console listeners separate from the data plane | ✅ | configuration | console split | |
| TLS between services (gateway, meta, OSD, block) | ✅ | mTLS on every gRPC hop, each call checked against the role its caller's certificate names; meta's admin port over it, membership changes for role admin only, its probes and status answered without a client certificate; client-facing TLS is at the ingress. Off by default: without it the internal ports are unauthenticated, and the chart's NetworkPolicies keep them to IOLake | mTLS | meta |
| NBD access control | ✖ | NBD has no authentication; exports are not listed, and the port must be reachable only from the machines that attach volumes | configuration |
Block¶
Preview: the crash tests under a filesystem and a database are the gate for production.
| Feature | Status | Notes | Guide | Design |
|---|---|---|---|---|
| Volumes over NBD and gRPC: read, write, flush, trim | 🟡 | feature reference | block storage | |
| Thin provisioning | 🟡 | feature reference | block storage | |
| Write journal (fsynced before the acknowledgement), replay | 🟡 | block write path | ||
| Snapshots, clones (shared stripes) | 🟡 | feature reference | block storage | |
| Volume metadata in meta (the gateway's disk holds only the journal) | 🟡 | block storage | ||
| Repair of block stripes | 🟡 | block storage |
File¶
Not built. Files are reached through S3 today; NFS on the same core is on the roadmap.
Platform¶
| Feature | Status | Notes | Guide | Design |
|---|---|---|---|---|
iolake-aio: meta, OSDs and the gateway in one process |
✅ | quickstart | ||
| Clean shutdown on SIGTERM | ✅ | Every service finishes its work in flight and exits 0; the gateway leaves the load balancer first, and delivers its audit spool before it exits | upgrades | |
| Linux binaries (x86_64, arm64), macOS for development | 🟡 | The Linux binaries need glibc 2.28 or later and no other library; each release's are run on Ubuntu 22.04, Debian 12, Rocky Linux 9 and Amazon Linux 2023 before it is published. iolake-install sets a host up under systemd (meta and the gateway by the chart's flags, an OSD per disk) and forms meta's Raft group, tested end to end on one host and on three |
Linux | |
| Container image (amd64, arm64) and Docker Compose cluster | ✅ | Docker | ||
| Helm chart | 🟡 | Production defaults: 4+2 over 6 OSDs (it refuses to render with fewer OSDs than the code needs), 3 metas, 2 gateways, auth on, disruption budgets. Placement constraints, NetworkPolicy, probes that ask meta's Raft and the OSDs' disks, raw block OSDs, failure domains from node labels, existing Secrets (SSE key kept on uninstall), a block gateway of one pod; a production example. Meta HA tested on kind in CI; installs, upgrades and failure-domain labels checked on kind, raw block volumes and NetworkPolicy beyond kind's not yet on a real cluster | Kubernetes | |
| Web console (operator and tenant) | ✅ | feature reference | console split | |
Admin CLI (iolakectl) |
✅ | CLI | ||
| Meta backup and restore | ✅ | iolakectl meta export backs up every table of meta's state from a live cluster, checked as it downloads; iolake-meta restore makes a new group from it, which lists again the objects written after it. A survivor of a meta majority lost for good can lead alone (iolake-meta force-single-voter), the last resort. Meta's own changes after the backup, and objects packed after it, are not recovered |
failure recovery | meta |
| Management SDKs (Go, Python) for the whole admin API | ✅ | SDKs | ||
| Agent skills for coding agents (Claude Code, Codex, opencode, pi): deploying and running a cluster, building on the APIs | ✅ | Carried by iolakectl (iolakectl skills install), or a Claude Code plugin from the repo |
agent skills | |
| Prometheus metrics (S3, Iceberg, replication, I/O) | ✅ | metrics | ||
| Grafana dashboards, alert rules | 🟡 | The docker-compose monitoring stack only | monitoring | |
| Heap profiles of any service with standard tools | ✅ | jemalloc's profiler built in, off until turned on; read with jeprof |
profiling | |
| Performance check on every change | ✅ | Release builds of a change and its base on one runner (in the merge queue, a dedicated one: a fresh container per job, no reach into its host's network); a drop in any cell fails it | performance | |
| Production guide: sizing, alerts, runbooks | ✅ | production | ||
| Hardware design and sizing guide, with a calculator | ✅ | From capacity and performance to a bill of quantities; drive failures, replacement time and hot spares | hardware sizing | |
| Documentation site, a version per release | ✅ | iolake.dev |