Skip to content

0.1.0

The first public release of IOLake: S3-compatible object storage in Rust, on one storage core (erasure coding, failure-domain-aware placement, repair, metadata on Raft), with data lake catalogs and block volumes on the same core. Apache-2.0, every feature.

Status: release candidate 0.1.0-rc.4, 2026-11-05 (0.1.0-rc.3: 2026-10-23; 0.1.0-rc.2: 2026-10-21; 0.1.0-rc.1: 2026-10-18). What remains before 0.1.0 is final is in the roadmap.

An early release

0.1.0 is early, and says so in its number. What that means:

  • Tested for production: S3 and the storage core. The evidence is below: ceph s3-tests, a 48-hour soak under faults, a chaos suite on six machines, and power-cut replay of the OSD's disk writes.
  • Previews: the data lake APIs and block volumes. Built and tested on their own, but not yet with real engines (Spark, Trino, pyiceberg) or under a filesystem and a database. File access is not built.
  • The promise that holds from 0.1.0 on: adjacent releases upgrade one node at a time, under traffic, with every object readable after; the release candidates do too, each to the next (0.1.0-rc.1 to rc.2, rc.2 to rc.3, and the last one to 0.1.0) (upgrades). An acknowledgement means the data is on stable storage, in every release.
  • What may change between 0.x releases: the admin API, the CLI and the console. A change to either is in the notes of the release that makes it.

Every feature and how far it has been proven is in features.md.

What ships

S3

  • The S3 API that applications use: buckets, objects, multipart uploads, copies, ranges, conditional requests and writes, checksums, tagging, versioning, Object Lock, lifecycle expiration, CORS, presigned URLs and POST uploads (S3 API).
  • S3 Select: SQL over CSV, JSON and Parquet objects, compressed or not, answered in AWS's event stream; the object is read in ranges and parsed as it arrives, Parquet by the columns the query names, so memory doesn't grow with the object (S3 API).
  • Gateway-side grep: ?grep on an object or a prefix streams matches as NDJSON with byte offsets, so an agent can fetch the context around a match with a range read instead of downloading the file (grep).
  • Audit: one event per request, with the request ID the client got, to webhooks, a file, stdout and each tenant's own webhooks; fsynced to a spool on the gateway before delivery, so a crash or a receiver outage loses nothing; on by default for the system, into an internal bucket (audit).
  • Bucket logging in the S3 access-log format (bucket logging).
  • Bucket replication between IOLake clusters: new writes as they happen, a resync of existing objects that resumes where it stopped, and a bandwidth limit (replication).
  • Quotas of bytes and objects per tenant and per bucket.

Core

  • Raw-disk OSDs: O_DIRECT, a write-ahead log, io_uring in Linux release builds (pread, with a warning, on a host that denies io_uring, such as a container under docker's default seccomp profile). Each shard is fsynced, then its location record, before a PUT is acknowledged.
  • An OSD's disks are known by the ids in their superblocks, not by their paths or order: an OSD missing one of its disks, or given a disk not its own, doesn't start, and a blank disk replaces a missing one. Every read and scrub checks that the block is the shard its location names.
  • Erasure coding (Reed-Solomon 4+2 by default) with ISA-L on x86_64 and arm64 alike: one code on every build, so amd64 and arm64 nodes mix in one cluster, and a pool takes only the profiles every loss of m shards decodes (data protection). Each shard's checksum is kept with the object and checked on every read and repair, and a data shard a read rebuilds is checked before it is served.
  • A pool's protection scheme is checked when the pool is made or changed, and meta's default scheme when it starts: a profile with no parity (1+0; one unprotected copy is replication of 1), an LRC whose data shards don't split evenly into its groups or that has no local or global parity, a replica count of 0, or more than 255 shards is refused with what is wrong with it (data protection).
  • LRC pools take objects of any length, PUT or multipart, coded as LRC stripes (not yet through the full end-to-end suite or a soak). An LRC stripe survives every loss of its global parity count, more by where they fall: LRC 6+2+2 any 2 shards and 88% of the ways of losing 3 (what LRC survives).
  • Every stripe is written with the same rules, whatever writes it: a PUT, a multipart upload's part, a pack of small objects, a block volume's chunk. It is acknowledged once k + 1 shards are on disk (two copies in a replicated pool, all but g − 1 shards in an LRC one), counted on distinct OSDs, and answered 503 short of that (507 when the OSDs are full); every shard carries its placement group's epoch, and one placed under an old epoch is placed again. A multipart part in a replicated pool was acknowledged with one copy, an LRC stripe with k + 1 shards it might not decode from, and a block chunk could count two shards on one OSD twice (writing a stripe).
  • Object metadata written at a majority of the object's OSDs and read from a quorum: writes, overwrites and deletes go on with one or two OSDs down.
  • Placement groups across failure domains (host by default; rack, zone and more if declared), each with an epoch committed before any write uses it; recovery, repair, backfill and drain by placement group; a drive lost for good re-protected without an operator.
  • Small objects: their shards travel with their metadata in one round trip, and many are packed into one stripe.
  • Meta on Raft (openraft) on three or more nodes, any node serving, with its own log: a meta node is back within a second of a power cut, and an OSD of 10 M shards serves 17 s after one. The leader answers only while a majority confirms it still leads: one cut off from its followers stops answering, rather than serve what may have changed elsewhere.
  • Rolling upgrades under traffic, new formats gated until every node runs the new release.
  • Bucket listings seek to the continuation marker and past each common prefix, so a page costs the same at the end of a large bucket as at its start.

Security

  • Users, keys, groups, policies and roles per tenant, through the admin API and the AWS IAM and STS APIs (aws iam, aws sts, boto3, Terraform); a tenant is an account, its admins its root (IAM).
  • Disabling a tenant refuses every identity of it within 15 s: its users' keys and presigned URLs, its STS and console sessions, and Delta Sharing reads of its shares (disabling a tenant).
  • STS from an OIDC token, with an identity provider per scope: the operator's for system roles, each tenant's for its own.
  • Console sign-in through the same providers: a sign-in is the user created for the token's issuer and subject, a tenant's provider signs in to that tenant only, and only the operator's provider and the system admin's own keys give an administrator's session (signing in to the console). Format level 9.
  • Anonymous access and Block Public Access at bucket, tenant and cluster level, new buckets blocked; named endpoints and client addresses (aws:SourceVpce, aws:SourceIp, trusted proxies) in policies (public access).
  • SigV4 in the header, in a presigned URL and on streaming uploads: the body is held to the hash it was signed with, and each streaming chunk and signed trailer to its signature (authentication).
  • SSE-S3, SSE-KMS with the built-in KMS, SSE-C.
  • mTLS on every gRPC hop between services, with a certificate per role and each call checked against its caller's role; meta's admin port over the same certificates, changing Raft membership for role admin only, while /status and the probes' /health and /ready answer without a client certificate; iolake-install cluster init takes an admin certificate (mTLS). NetworkPolicies, on by default in the chart, keep the internal ports to IOLake's own pods; the all-in-one keeps them on loopback.

Data lake (preview)

  • Iceberg REST Catalog: warehouses, namespaces, tables, commits, policies, vended credentials (Iceberg API).
  • Unity Catalog API at /api/2.1/unity-catalog/*: catalogs, schemas, tables, functions, volumes, models and versions; policies at each level; row filters and column masks bound to SQL functions, which engines read and apply; a backing bucket per catalog that refuses anything but vended credentials, so a permanent key can't bypass a filter; Keycloak groups mapped to IAM groups by name (feature reference).
  • Delta Sharing: shares, recipients, presigned queries over Delta (JSON log) and Iceberg tables (feature reference).

Block (preview)

  • Volumes over NBD and gRPC: thin provisioning, snapshots and clones that share stripes, a write journal fsynced before each acknowledgement, volume metadata in meta, repair of block stripes.

Platform

  • iolake-aio, the whole cluster in one process; binaries for Linux x86_64 and arm64 (and macOS for development); a multi-arch image (quickstart).
  • A Helm chart whose defaults protect data and require auth: 4+2 erasure coding over 6 OSDs (it refuses to render with fewer OSDs than the code needs), 3 metas, 2 gateways, disruption budgets for all three. A values file can express a production deployment: placement (node selectors, affinity, tolerations, spread constraints), NetworkPolicies, raw block volumes for OSDs, each OSD's zone and rack read from its node's labels, existing Secrets for the SSE key, OIDC and Delta Sharing (a generated SSE key is kept across upgrades and on uninstall), and probes that ask meta whether its Raft runs and has a leader and the OSDs whether their disks are healthy. The block gateway is a StatefulSet of one pod over its journal; helm install --wait completes. The kind setup sets its own development values (Kubernetes).
  • The Linux binaries need glibc 2.28 or later and no other library: PCRE2 is linked in and TLS loads no OpenSSL. Each is run on Ubuntu 22.04, Debian 12, Rocky Linux 9 and Amazon Linux 2023 before the release is published (Linux).
  • iolake-install sets a Linux host up under systemd: units that start meta and the gateway with the flags the Helm chart passes, an OSD per disk with its own config file and ports, and cluster init, which forms meta's Raft group over the admin ports. Every binary's default ports are the cluster's (meta 9100, OSD 9200); meta and the gateway take flags only. Tested end to end: one host, then three, run from the units the installer wrote, serve S3 (Linux).
  • Every service stops cleanly on SIGTERM: it finishes its work in flight and exits 0, and the gateway leaves the load balancer, drains, and delivers its audit spool first, so a rollout loses no request or audit event (upgrades).
  • The web console (operator and tenant), iolakectl, and Go and Python SDKs for the whole admin API (SDKs).
  • The admin API keeps its references whole and says why it refuses: an OIDC provider a tenant names can't be deleted (409, naming the tenants), and a tenant can't name one that doesn't exist; a tenant's audit target other than a webhook is refused saying what a tenant can use; a tenant admin lists its own tenant's Iceberg warehouses. A fresh cluster's iolakectl upgrade status has every node reported, its meta included, from the start. iolakectl replication settings sets the hosts tenants' replication targets may be on, and pool create --pg-count says how meta sizes a count left out (CLI, IAM).
  • Prometheus metrics; a production guide with sizing, alerts and runbooks; a hardware sizing guide with a calculator (production, hardware sizing).

Changes since 0.1.0-rc.3

Changed

  • ObjectIO is now IOLake. The code is at io-lake/iolake and these pages at iolake.dev, and every name you run or read says IOLake: the binaries (iolake-aio, iolake-meta, iolake-osd, iolake-gateway, iolake-block-gateway, iolake-install), the CLI (iolakectl), the systemd units, the environment variables (IOLAKE_*), the metrics (iolake_*) and alerts (IOLake*), the image (ghcr.io/io-lake/iolake), the Helm chart (oci://ghcr.io/io-lake/charts/iolake), the SDKs (pip install iolake-sdk, import iolake; github.com/io-lake/iolake-go-sdk) and the agent skills. What a cluster stores, and what its nodes say to each other, keeps its names, so an rc.3 cluster upgrades in place, one node at a time (below): data and configuration stay in /var/lib/objectio and /etc/objectio, meta and the gateway still run as the user objectio, the CLI keeps its profiles in ~/.objectio/config, the chart's system audit bucket is still objectio-audit, ARNs keep arn:obio: and arn:objectio:, and a certificate still names its role as spiffe://objectio/<role>. The releases up to 0.1.0-rc.3 stay where they were published, under the ObjectIO names (ghcr.io/object-io/objectio, objectio-aio-<tag>-<platform>).

Added

  • The console is redone. Pages are grouped by job, every page has the same layout and a top bar that shows whether the cluster's data is protected, rebuilding or at risk, and ⌘K jumps to any page. The overview leads with data protection and a list of what needs attention; a new Cluster health page shows placement groups, repair, drains and every drive by rack and host; buckets have a details panel and the object browser is inside each bucket; Iceberg and Unity tables share one catalog browser. Light and dark themes (web console).

Upgrading a 0.1.0-rc.3 cluster

One node at a time, as any upgrade (upgrades): rc.3 nodes and nodes of this release run side by side, since nothing stored or sent between nodes is renamed. What changes is what you run, and what reads the cluster's names:

  • The binaries. Install iolake-meta, iolake-osd, iolake-gateway, iolake-block-gateway, iolake-aio and iolake-install (release downloads are iolake-aio-<tag>-<platform>), and iolakectl in place of obioctl. Their flags are unchanged. An all-in-one keeps its data: start iolake-aio with the --data it had (the guides' example is now $HOME/.iolake; it was $HOME/.objectio).
  • The systemd units are iolake-meta, iolake-gateway and iolake-osd-<i>, where objectio-install wrote objectio-*; the data directories, OSD files and user are the same. Write the new units from the old ones, then switch each service in the procedure's order (meta followers, the leader, the OSDs, the gateways):
cd /etc/systemd/system
for u in objectio-*.service; do
  n="iolake-${u#objectio-}"
  sed 's/objectio-/iolake-/g' "$u" | sudo tee "$n" >/dev/null
  if [ -d "$u.d" ]; then                     # drop-ins (mTLS), with their variables
    sudo mkdir -p "$n.d"
    for f in "$u.d"/*.conf; do
      sed 's/OBJECTIO_/IOLAKE_/g' "$f" | sudo tee "$n.d/${f##*/}" >/dev/null
    done
  fi
done
sudo systemctl daemon-reload
# then, for each service in turn, once its new binary is in /usr/local/bin:
sudo systemctl disable --now objectio-meta && sudo systemctl enable --now iolake-meta

When every service runs under its new unit, remove the old ones (/etc/systemd/system/objectio-*.service and their .d directories) and the old binaries. - Environment variables are IOLAKE_*, and a service ignores the old OBJECTIO_* names. Rename them wherever you set them: units and drop-ins, Compose files, the chart's values, scripts. The one that matters most is the SSE master key in /etc/objectio/gateway.env: a gateway that doesn't find IOLAKE_MASTER_KEY can't read the objects it encrypted. Before switching a host's gateway: sudo sed -i 's/^OBJECTIO_/IOLAKE_/' /etc/objectio/gateway.env. - Metrics and alerts. Every metric objectio_<name> is now iolake_<name>, and every alert ObjectIO<Name> is IOLake<Name>. Update your dashboards, alert rules and recording rules (for a file, sed -i 's/objectio_/iolake_/g; s/ObjectIO/IOLake/g'); while old and new nodes run side by side, a node exports the names of its release. The chart's own dashboards and rules come with the upgrade. - The image and the chart. The image is ghcr.io/io-lake/iolake (a Compose file's or a pod's command names the new binaries) and the chart oci://ghcr.io/io-lake/charts/iolake. An upgraded Helm release sets nameOverride=objectio to keep its resources' names, and with them its volumes and its generated SSE key (Kubernetes). - The CLI is iolakectl, with the same commands; it reads the same profiles, and the variables it reads are IOLAKE_ENDPOINT, IOLAKE_ACCESS_KEY, IOLAKE_SECRET_KEY, IOLAKE_PROFILE and the rest of IOLAKE_*, and IOLAKECTL_REMOTE_SECRET_KEY for OBIOCTL_REMOTE_SECRET_KEY (CLI). - Clients: S3 CreateBucket takes a bucket's pool in x-iolake-pool (it was x-objectio-pool); the SDKs are iolake-sdk (Python, import iolake) and github.com/io-lake/iolake-go-sdk (Go, package iolake), reading the same IOLAKE_* variables. Console sessions are signed in again once their gateway is upgraded: the session cookie is now iolake-session. Agent skills are iolake-*: iolakectl skills install adds them, and the objectio-* copies an earlier CLI installed can be deleted (agent skills).

Fixes

  • A cluster with one meta node no longer answers writes 503 on a slow disk. Meta gave up confirming its own leadership after a second, and with one voter that confirmation waits for the log write in progress, so a disk sync over a second turned a write into "the metadata service is unavailable". The only voter now waits for it; clusters with more voters keep the one-second limit (meta).
  • The console reads and changes a bucket's versioning, object lock, encryption, lifecycle and replication. It asked the S3 API for them, which refuses a console session, and showed them as off; the admin API now has them, for the system administrator and the bucket's tenant administrators (bucket settings).
  • PyIceberg with SigV4 works as the docs configure it. It signs with botocore's generic SigV4 signer, over the hash of the body without sending it in x-amz-content-sha256, and its first call was refused. The admin API, the Iceberg REST catalog and Unity Catalog now hash a body the signature was made over, up to 2 MiB, and refuse a body changed after signing; a larger one is refused with 413 unless its hash is sent. The S3 API is unchanged (the signed body).
  • The S3, Iceberg and Unity request-duration histograms counted each request in its buckets twice, so a bucket could pass +Inf and every percentile read from them came out low.
  • Console form labels are tied to their fields, so a screen reader names them.

Changes since 0.1.0-rc.2

Added

  • Meta can be backed up and restored. iolakectl meta export takes a backup of meta's whole state from a live cluster; iolake-meta restore makes a new meta group from it after every meta node's data is lost, and the objects written after the backup are listed again. When a majority is lost for good and one node survives, iolake-meta force-single-voter --others-lost-for-good makes it the only voter, the last resort (failure recovery).
  • The gateway's memory is bounded under any load. A GET streams the object stripe by stripe instead of building it whole in memory; request bodies and GET read-ahead come from a memory budget (--memory-budget, by default half the container's memory limit); a request that doesn't fit, or one over the in-flight limit (--max-requests), gets 503 SlowDown; a client that stops sending or reading is cut off. A body over the limit is now 400 EntityTooLarge, as S3 answers, not 413 (production).
  • A single PUT and an upload part go up to S3's 5 GiB, and an object to 5 TiB; both were 100 MiB at most, so an object was about 976 GiB at most. A body over 16 MiB is streamed to the shards stripe by stripe as it arrives, never held whole in the gateway, which holds a few stripes of it (about 48 MiB with 4+2) from its memory budget whatever its size. It is committed only once all of it has arrived and passed its signed payload hash, chunk signatures, Content-MD5 and checksum; one that fails, or a client that goes away mid-body, stores nothing and has its shards freed. A copy that changes the encryption streams too, from a source of up to 5 GiB. Larger bodies get 400 EntityTooLarge before they are read; a completion over 5 TiB gets it too (large uploads).
  • Virtual-hosted addressing. With the gateway's S3 domain set (--s3-domain s3.example.com; gateway.s3Domains in the chart), a request to photos.s3.example.com/key addresses bucket photos, as on AWS: objects, bucket operations on /, presigned URLs and policies all see the bucket from the host, and path-style keeps working beside it. A request whose bucket can't be told for certain is refused instead of misread: a bucket operation on / from a host that names no bucket (GET /?list-type=2 used to come back as the caller's bucket list) is 400 InvalidRequest, and a host under the domain that isn't a bucket name is 400 InvalidBucketName. Wildcard DNS and a wildcard certificate for the domain are the operator's (virtual-hosted addressing).

Upgrading a 0.1.0-rc.2 cluster

  • Format level 8, 0.1.0-rc.1's, is the lowest a release runs at, and the code for the levels below it, which only development builds ran, is gone. Upgrades go one release at a time, each release candidate to the next: an rc.1 cluster goes to rc.2 first, and is finalized, before this release. A node of this release refuses to start in a cluster below level 8, saying it is a development build's (upgrades).
  • Upgrade once an OSD has registered. A meta node of 0.1.0-rc.2 keeps its Raft log in its database until the cluster's first OSD registers; one upgraded before that stops at start-up, saying so, rather than start with an empty log.
  • The gateway's --heal-interval-secs is gone, with the heal queue it set the pace of: a write or delete that left a metadata copy behind marks its key's placement group, whose peering and recovery bring the copies in line, as they already did for every key placed through one. Take the flag out of your units and scripts before upgrading a gateway: a binary refuses a flag it doesn't have. The metrics objectio_gateway_heal_total, objectio_meta_repair_listings_restored_total and objectio_meta_repair_shards_moved_total went with what they counted; the repair walk now covers block chunks and packs, which no placement group holds (configuration).
  • Flags that did nothing are gone, and a binary refuses to start with a flag it doesn't have: meta's --peers (the Raft group is formed over meta's admin port) and the gateway's --osd-endpoint (the gateway finds the OSDs through meta). Take them out of your units and scripts before upgrading; the chart no longer passes --peers (configuration).
  • An OSD config with a key the OSD doesn't know stops the OSD, as one that doesn't parse already did. Before upgrading an OSD, remove from its file any key it never read: node_id and a [cache] table, which 0.1.0-rc.1's objectio-install wrote (OSD configuration).

Fixes

  • The container images' default commands start their service: the meta, OSD and gateway images passed --bind, which none of them has.
  • A lost meta leader is replaced in about 4 s, not 7–12 s. The followers stand for election after 3–4.75 s without a heartbeat, where they waited 6–8.75 s, and a round lost to a candidate whose log was behind costs 1–2.75 s more, where it cost up to 4.75 s: with three meta nodes under writes, killing the leader paused writes for 3.6–4.4 s, against 6.7–8.7 s (12.5 s seen with a lost round). --raft-election-ms now defaults to 1000, with a 2 s leader lease; give every meta node the same value. A follower still tolerates 3 s of silence before standing, twice what caused elections under load with no fault, and the leader still answers only within a quorum confirmation that no other leader can be elected inside. A cluster mixing upgraded and older nodes is safe (meta HA, configuration).
  • A meta leader that voted for another node, or stalled, no longer answers a read from before the new leader's writes. openraft lets a leader of more than 2 s grant its vote to a node of a higher term whose log is as long as its own (a follower back from a pause or a one-way partition, with nothing written meanwhile); that node then led with this vote and its own, without waiting out the other follower's lease, and committed writes, while the old leader could go on answering for up to 500 ms from its last quorum confirmation, until its own state showed it had stepped down. And every read waiting on a confirmation round was answered when the round succeeded, however long after it started: a leader that stalled for seconds with the followers' acks in hand answered, on waking, reads sent after the new leader's writes. Now answering a vote request voids the confirmation and the round under way, and a confirmation answers a read only inside its 500 ms, or when its round started after the read arrived; any other read waits for a new round, or fails UNAVAILABLE, which clients retry (meta).
  • No OSD holds two shards of one stripe. The repairer rebuilt a missing shard on the OSD its position is placed on now even when that OSD held another shard of the stripe (a block chunk or a pack written before its placement group changed), so one lost disk cost the stripe two shards; such a shard now stays missing, the object degraded, until its position has an OSD of its own. A placement group's stand-ins and moves back to its up set never put two of its positions on one OSD: a member back from down could become the stand-in for another position, leaving up naming it at two, and recovery then moved both onto it. And a key placed without a placement group (one written before format level 7) whose shards had too few OSDs was placed with the least-loaded OSD named twice; it is now refused, 503 ServiceUnavailable, to retry (topology).
  • A delete that meets a placement change is placed again, as a PUT already was: when an OSD was set out between placing a delete and sending it, every copy refused it as stale and the delete answered 503.
  • A TLS handshake never waits forever, on either side. Under mTLS, meta, the OSDs and the block gateway disconnect a peer that hasn't completed its handshake 10 s after connecting, as meta's admin port already did, where a peer that connected and said nothing held its connection, and a task, for as long as it liked; a refused peer is logged. A node connecting to another gives up when its connect timeout is out (5 s by default), the handshake included, where only the TCP connection was bounded: a peer that took the connection and then answered nothing, a frozen process for one, held the call forever. And a gRPC port goes on accepting whatever a peer does (mTLS).
  • A disk's failed reads, writes and flushes are counted (iolake_disk_errors_total); one that has failed makes the OSD's /health answer 503 (monitoring).
  • Every service image checks its own health (Docker). The block gateway serves /health on its metrics port.
  • Block calls that aren't built answer UNIMPLEMENTED: QoS settings, volume stats, and the OSD, cluster and I/O-trace metrics calls. They answered OK, or zeros, and did nothing.
  • A Delta share refuses an Iceberg table, which it served as files no client could read (Delta Sharing).
  • A full filesystem under an OSD answers 507 StorageFull, not 503. When the filesystem under a file-backed disk or the OSD's state directory ran out of room (ENOSPC, EDQUOT), a write got 503, which clients retry, and the OSD's metadata log took no writes until the OSD was restarted. Now nothing of the write is kept, reads go on, the disk isn't reported unhealthy, and writes resume once there is room. A write whose OSDs are partly full and partly unreachable is 503 when the unreachable ones could still make its quorum, and 507 only when the full ones keep it short (full disks).
  • A meta node whose disk fills refuses writes, 507 StorageFull, instead of exiting. A write the filesystem under meta's data refused stopped the node's Raft, and the node exited, to be restarted on the same full disk; with the leader's or a majority's disk full, the cluster took no writes and, while nodes restarted, answered nothing. Meta now keeps 1 GiB free (--min-free-mib): below it the leader refuses writes that need meta (creating a bucket, IAM, multipart uploads, settings), while reads and object PUTs go on; a follower falls behind and catches up; a snapshot waits for room. Writes resume once there is room, with no restart. If something else on the filesystem takes the last bytes, a Raft log write waits for room rather than stop the node, and a failed database write still stops it, to be started again with nothing acknowledged lost. Meta reports its free space (iolake_meta_disk_free_bytes, iolake_meta_disk_full, /status), and the chart alerts on it (meta's disk filling).
  • GetBucketLocation answers the bucket's region, the gateway's --region (us-east-1 as the empty constraint, as S3 answers it), in path-style and virtual-hosted addressing alike. GET /bucket?location answered with the bucket's listing. It is authorized as s3:GetBucketLocation, no longer as s3:ListBucket: a policy that allows only the listing no longer allows it, as in S3. A GET on a bucket that names a sub-resource the gateway doesn't implement (?session, ?metadataTable, or any other without a value, besides those already refused) is 501 NotImplemented, not a listing (S3 API).
  • An OSD that stops answering holds requests for about 3 s, once. A gateway took an OSD as up again when it accepted a connection, which the kernel of a frozen process (a stopped container, a hung host, one cut off without a reset) still does. Without mTLS, the chart's default, every request that touched a frozen OSD then opened a new connection to it and waited 5 s on that (a DELETE 15 s, a HEAD of a missing key 10 s) for as long as it stayed frozen, though the other OSDs held every quorum; with mTLS the first requests waited 5 s, and every DELETE 3 s. The gateway now probes an OSD with a call it must answer, and abandons a call that has run 2 s to an OSD that doesn't: the first requests to meet a frozen OSD wait about 3 s, those after them not at all. Version listings and the shard frees after a DELETE are bounded the same way. Nothing is acknowledged with fewer copies than before (an OSD that stops answering).

Changes since 0.1.0-rc.1

Upgrading a 0.1.0-rc.1 cluster

One node at a time, as any upgrade, then obioctl upgrade finalize (upgrades). 0.1.0-rc.2 is format level 10; 0.1.0-rc.1 is 8. Before finalizing:

  • Console sign-in through an identity provider is refused until the upgrade is finalized; sign-in with an access key works throughout (IAM).
  • Under mTLS, every certificate must name its role once the upgrade is finalized. The chart issues and rolls the per-role certificates itself; with certificates of your own, reissue them with roles first (mTLS).
  • Console sessions and temporary credentials (STS sessions, vended credentials) issued by an 0.1.0-rc.1 gateway aren't accepted by an upgraded one: sign in again, and get new ones.
  • On Kubernetes, the chart's defaults changed (a 4+2 pool over six OSDs, authentication on, NetworkPolicies on), and three changes need a step before helm upgrade (upgrading from 0.1.0-rc.1).
  • On Linux hosts, meta and the gateway take flags only, no --config; objectio-install writes their units that way (Linux).

Security

  • A signed request's body is checked against what was signed: the payload hash in the header, each streaming chunk's signature and the signed trailer (authentication). Session tokens are no longer logged.
  • Console sign-in through an identity provider is the user bound to the token's issuer and subject, never a user found by name, and a tenant's provider signs in to that tenant only.
  • Disabling a tenant refuses every identity of it within 15 s: keys, presigned URLs, STS and console sessions, Delta Sharing reads (disabling a tenant).
  • Under mTLS, each call between the services is checked against the caller's role, named in its certificate; meta's admin port takes membership changes only from an admin certificate, and serves HTTPS (mTLS). NetworkPolicies, on by default in the chart, keep the internal ports to IOLake's own pods.
  • A metadata write too large to apply is refused before it is committed. One such write, which any S3 user could send, stopped every meta node.
  • The data lake APIs authorize as S3 does:
  • A credential is required. The Iceberg REST catalog and the Unity Catalog API refuse a request with no credential, or one that doesn't check out, with 401; they are open only under --no-auth.
  • No policy means administrators only. A request is granted by an Allow in a policy (and, on Unity, the caller's IAM policies), or goes ahead for the warehouse's or catalog's administrators: before, any authenticated caller was let through when no policy said otherwise. Tenants see only their own warehouses and catalogs. (Iceberg API)
  • Vended credentials are read-only unless the caller may commit to the table, are confined to the warehouse's bucket, and stay inside the warehouse's (or catalog's) tenant. Unity's external tables and volumes, like Iceberg tables, must lie inside their catalog's bucket. Session tokens now carry their tenant, so temporary credentials issued by an 0.1.0-rc.1 gateway (STS sessions, vended credentials) are not accepted once a gateway is upgraded: clients get new ones.
  • Admin API: the Delta Sharing management routes moved from /_admin/delta-sharing/* to /_admin/shares/{share}, /_admin/shares/{share}/tables and /_admin/recipients, behind the admin API's authentication; a share takes tables of its own tenant only, unless the system admin adds them. The warehouse routes on the Iceberg API (/iceberg/v1/warehouses) are gone: /_admin/warehouses manages warehouses, and refuses to delete one whose bucket holds objects.
  • S3: a request whose bucket, policies or attached policies meta can't read is refused with 503 (retry), rather than decided without them.

Fixes

Data read and written correctly:

  • An encrypted object larger than one stripe (16 MiB at 4+2), stored by a single PUT, read back whole with wrong bytes after the first stripe. The stored data was right, and ranged and multipart reads were too.
  • A shard rebuilt by erasure decoding is verified before it is served, against its recorded checksum or a parity shard the decode didn't use. Builds with different codecs could serve wrong bytes with a 200; every build now codes with ISA-L, on x86_64 and arm64.
  • An OSD knows its disks by their ids. Started without one of its disks, an OSD renumbered the others, and could overwrite live shards.
  • The OSD's metadata log, the block journal and meta's Raft log stop on a failed write or sync, and refuse to open over damage rather than cut it off as a torn tail.
  • Every stripe is written by one writer, with one quorum, fence and answer for every scheme; an upload part in a replica pool needs the replica quorum, as a PUT does.
  • LRC writes of any length succeed; a pool with no parity, or a scheme the codec can't take, is refused when it is made.

Available when nodes fail:

  • A meta leader cut off from its followers stops answering once no quorum confirms it leads, rather than serving its own state.
  • A meta node that stops answering mid-call fails the call in about 5 s (HTTP/2 keepalive on every meta connection); before, a call could wait on a frozen node for good.
  • A gateway is ready only once it reaches a write quorum of OSDs for the default pool.
  • SIGTERM stops every service cleanly: the gateway drains, meta stops Raft before its last checkpoint, the block gateway syncs its journal.
  • A port a service can't bind fails its start, rather than leaving it running without its metrics or admin port.
  • A node that joins by snapshot numbers config changes as the leader does.
  • A bucket listing seeks to its marker: paging through a bucket reads each row once, not every row before the page again.
  • The Linux release binaries need only glibc 2.28, and are tested on Ubuntu 22.04, Debian 12, Rocky 9 and Amazon Linux 2023 before they are published.

How it was tested

On 0.1.0-rc.4's code: the first row, on every change, and ceph s3-tests on Linux x86_64. The soak, the chaos suite, s3-tests on arm64 and the performance comparison ran on 0.1.0-rc.1's.

Test Result
Unit and end-to-end tests against real processes, a meta HA test on Kubernetes (kind), a rolling upgrade under traffic, the crash-consistency test, and a performance check, on every change; the data-path end-to-end tests also on binaries built with the release's features (io_uring). A test that can't run where it is run (no root, no previous release, no Mooncake build) fails, never passes untested Pass
ceph s3-tests, Linux x86_64 681 of 687 applicable pass; the other 6 are known gaps. No bugs
ceph s3-tests, Linux arm64 679 of 687 applicable pass; the other 8 are the same 6 gaps and 2 tests that fail only beside others (each passes alone). No bugs
Soak: 48 hours on a lab host, faults every half hour or so Pass: 84 faults (power cuts, meta leader kills and partitions, disks pulled, drives lost for good), 7.02 M acknowledged operations, 1.50 TB written; every key audited at the end, nothing lost, no space leaked
Chaos: six machines, writes through two gateways, six faults Pass: 81,436 acknowledged writes intact after every fault; longest write pause 13.2 s (the meta leader partitioned); afterwards any two OSDs could be stopped with every object readable
Crash consistency: the OSD's disk writes recorded and replayed up to random power-cut points Every acknowledged write survives
Performance Single host and six cloud VMs, side by side with RustFS and SeaweedFS

Known issues

  • Meta's memory grows under sustained load: from about 37 MB to 190–340 MB over the 48-hour soak, 498 MB at the peak. It doesn't grow with uptime alone; a heap profile is next.
  • A partitioned meta leader paused writes longer than before: 13.2 s in the chaos suite, against 5.4 s on 2026-10-04; within the suite's 60 s limit. Most of it was the election, which then began about 7 s after the leader's last heartbeat and now begins 3–4.75 s after it (below); the suite hasn't been run again since.
  • A write can fail with 503 when its host stalls: seen once in the test suite, on a busy CI runner, right after a test had written and read back 1.3 GB. A delete reached none of its six metadata copies within their 10 s deadline, all six on OSDs in the same process, and was answered 503, as it should be with no copy reachable; the client's retry succeeds. It doesn't happen locally. One OSD stopping, restarting or freezing under load fails no request (tested); an OSD that answers slowly but still answers is not yet handled (on the roadmap).
  • Rebuilding a replaced drive takes minutes even on a small cluster. With six OSDs and 4+2, a blank replacement disk lacks a shard of nearly every object. On the chaos VMs, filling one took 482 s; this was first reported as repair after a one-minute outage, which took 151 s there and takes under 30 s in the test suite. Each OSD recovers two placement groups at a time by default, so that client I/O keeps priority, and pg/backfills_per_osd raises it. Objects stay readable meanwhile, but until repair is done some have less redundancy than their pool's scheme gives (how long repair takes). Faster rebuilds are planned for 0.2.0.