Skip to content

IOLake — Feature Reference

What IOLake does today, and how. This page describes what is built. Every feature's status is in features.md, what is planned in the roadmap, and what each release added in its notes (0.1.0).

IOLake is a unified software-defined storage (SDS) platform: several storage paradigms on a single erasure-coded backend.

Paradigm Protocol Use Case
Object Storage S3 API Unstructured data, backups, media, ML datasets
Data Lakehouse Iceberg REST Catalog + Unity Catalog + Delta Sharing Analytics warehouses, three-level governance, Spark/Trino/Databricks-SDK queries, cross-org data sharing
Block Storage gRPC + NBD Databases, VMs, thin-provisioned volumes with snapshots

All share the same underlying erasure-coded storage layer, IAM, multi-tenancy, and web console. File access (NFS) is planned on the same layer.

1. Object Storage (S3 API)

Core S3 API

  • Bucket operations: CreateBucket, DeleteBucket, HeadBucket, ListBuckets
  • Object operations: PutObject, GetObject, HeadObject, DeleteObject, DeleteObjects (batch)
  • Multipart upload: InitiateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts, ListMultipartUploads
  • Range requests: HTTP Range header support (bytes=0-99, suffix ranges, open-ended ranges)
  • Copy object: Server-side copy via x-amz-copy-source header (metadata-only fast path when on same OSD)
  • S3 Select: SelectObjectContent on CSV, JSON and Parquet objects (GZIP/BZIP2 for CSV and JSON), CSV or JSON output
  • User metadata: x-amz-meta-* headers stored and returned
  • S3 error responses: Standard XML error format with S3 error codes

Bucket Versioning

  • PUT/GET /{bucket}?versioning — Enable, suspend, or query versioning state
  • Version-aware PUT: Generates UUID version ID when versioning enabled, returns x-amz-version-id header
  • Version-aware DELETE: Creates delete markers (new version with is_delete_marker=true) when no ?versionId specified
  • Version-specific DELETE: Permanently deletes a specific version via ?versionId=
  • ListObjectVersions: GET /{bucket}?versions returns all versions including delete markers
  • Dual-key OSD storage: Current version at m:{bucket}\0{key}, version history at v:{bucket}\0{key}\0{version_id}

Object Locking (WORM)

  • PUT/GET /{bucket}?object-lock — Configure object lock with default retention
  • Retention modes: GOVERNANCE (bypassable with header) and COMPLIANCE (immutable until expiry)
  • PUT/GET /{bucket}/{key}?retention — Per-object retention (mode + retain-until-date)
  • PUT/GET /{bucket}/{key}?legal-hold — Per-object legal hold (ON/OFF)
  • Delete enforcement: Objects under retention or legal hold cannot be deleted (403 AccessDenied)
  • Governance bypass: x-amz-bypass-governance-retention: true header allows deletion
  • Lock at creation: x-amz-bucket-object-lock-enabled: true on CreateBucket; or later, PutObjectLockConfiguration on a bucket with versioning enabled (otherwise 409 InvalidBucketState), as S3 allows. Versioning can't be suspended afterwards

CORS and browser uploads

  • CORS (?cors): preflights answered without credentials, headers on matching requests and their errors. S3 API § CORS
  • POST Object: S3 HTML form uploads (generate_presigned_post), SigV4 policy signatures, every policy condition, authorized as the signer and written as a PUT. S3 API § POST Object
  • GET response-* overrides (e.g. response-content-disposition on a presigned link), and objects keep Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Expires.

Lifecycle Management

  • PUT/GET/DELETE /{bucket}?lifecycle — S3's lifecycle configuration, checked as S3 checks it and read back as written. See S3 API § Lifecycle.
  • Expiration by Days or Date: unversioned buckets delete; versioned ones add a delete marker, as a DELETE would.
  • NoncurrentVersionExpiration (NoncurrentDays, NewerNoncurrentVersions): old versions expire by how long they've been noncurrent, keeping the newest N.
  • ExpiredObjectDeleteMarker: removes a delete marker only when it's the key's only version — never one with versions behind it.
  • AbortIncompleteMultipartUpload.
  • Filters: prefix, tags, object size, and <And>.
  • Safe by construction: every action goes through the S3 delete path, so Object Lock is honoured, the listing is kept and space reclaimed; every action is an audit event by lifecycle:<rule id>.
  • One scanner: gateways take a lease in meta, so one scans at a time (--lifecycle-interval-secs, default hourly).
  • Not supported: transitions (one storage class) — refused with InvalidStorageClass.

Bucket Policies

  • PUT/GET/DELETE /{bucket}?policy — IAM-style JSON bucket policies
  • Policy evaluation: Allow/Deny/ImplicitDeny decisions
  • ARN-based resources: arn:obio:s3:::{bucket} and arn:obio:s3:::{bucket}/{key}
  • Condition keys: s3:x-amz-server-side-encryption, s3:x-amz-server-side-encryption-aws-kms-key-id, aws:SecureTransport — populated from PUT/GET/DELETE request headers, evaluated at request time

Server-Side Encryption (SSE)

All three AWS S3 encryption modes are supported (SSE-S3, SSE-KMS, SSE-C). Encryption always happens before the erasure-coding stage so OSD shards only ever see ciphertext. Objects written without SSE remain plaintext; enabling SSE on a bucket does not retroactively encrypt existing objects. Ciphertext correctness is verified end-to-end in tests — grep for any plaintext marker on OSD shards returns zero matches when SSE is enabled.

  • PUT/GET/DELETE /{bucket}?encryption — Bucket default encryption configuration, AWS-compatible XML (ServerSideEncryptionConfiguration)
  • Per-request precedence: explicit x-amz-server-side-encryption* request header > bucket default > plaintext
  • ETag semantics: MD5 of plaintext for SSE-S3 / SSE-KMS (AWS-compatible). SSE-C uses the AWS multipart ETag format for multipart uploads
  • Response headers: x-amz-server-side-encryption, x-amz-server-side-encryption-aws-kms-key-id, x-amz-server-side-encryption-customer-algorithm, x-amz-server-side-encryption-customer-key-md5

SSE-S3 (AES256)

  • AES-256-CTR with a fresh per-object Data Encryption Key (DEK)
  • DEK wrapped with the gateway's service master key via AES-256-GCM and stored in ObjectMeta.encrypted_dek
  • Service master key loaded from IOLAKE_MASTER_KEY env var (base64 32 bytes); helm chart auto-generates one on first install and preserves it across helm upgrade
  • Range GETs and multipart uploads fully supported — each part/stripe uses its own IV so stripes decrypt independently

SSE-KMS (aws:kms)

  • Per-object DEK wrapped by a named KMS key (Key Encryption Key, KEK)
  • Local KMS provider: self-contained backend — KEK material is itself wrapped by the service master key before it reaches meta, so meta stores only encrypted blobs
  • Envelope encryption: client specifies KMS key via x-amz-server-side-encryption-aws-kms-key-id (full ARN or short id)
  • Encryption context: optional x-amz-server-side-encryption-context (base64 JSON) bound to the DEK wrap as AEAD associated data; decrypt fails if context differs
  • Multipart support: MPU picks the key at CreateMultipartUpload time; per-part stripe IVs

SSE-C (customer-provided keys)

  • Three headers: x-amz-server-side-encryption-customer-algorithm: AES256, -customer-key (base64 32 bytes), -customer-key-md5 (base64 MD5 of the key)
  • Key material is never persisted — object metadata stores only the algorithm marker + IV
  • MD5 binding verified on every PUT/GET/HEAD; wrong MD5 → InvalidArgument 400
  • Hard-blocked on warehouse buckets (iceberg-* and the configured warehouse bucket): query engines can't provide the customer key on every read, so writes are rejected at PUT rather than silently producing unreadable objects
  • Multipart upload: supported end-to-end. CreateMultipartUpload stores the customer-key MD5 in the MPU state; every UploadPart must resupply the same key, validated MD5-against-MD5 before encryption; CompleteMultipartUpload produces a standard AWS multipart ETag. The key itself never touches meta.

KMS Administration (/_admin/kms/*)

AIStor-compatible KMS action model. Resource ARNs use arn:obio:kms:::{key_id}; the wildcard arn:obio:kms:::* matches all keys.

Endpoint Action Description
GET /_admin/kms/status kms:Status Backend state (enabled, local, master-key configured)
GET /_admin/kms/version kms:Version Service version + protocol
GET /_admin/kms/api kms:API Supported endpoints
POST /_admin/kms/keys kms:CreateKey Create a new KMS key
GET /_admin/kms/keys kms:ListKeys List KMS keys
GET /_admin/kms/keys/{key_id} kms:KeyStatus Get a specific key
DELETE /_admin/kms/keys/{key_id} kms:DeleteKey Delete + drop cached KEK
  • Policy gating: every endpoint evaluates the caller's attached named IAM policies against (action, resource). The built-in admin user and system administrators' console sessions bypass policy evaluation; other console sessions are refused.
  • Dual auth on admin endpoints: SigV4 (for API clients) or console session cookie (for web UI). Cookie requests pass through a pre-router layer that validates SigV4 only when an Authorization header is present.

KMS Backend Selection

The gateway's KMS backend picks how SSE-KMS wrap/unwrap is performed.

Two configuration sources, meta wins: 1. Meta kms/config (console-writable) — the persisted config. Hot-swappable at runtime via PUT /_admin/kms/config with no gateway restart. 2. --kms-backend CLI flag + env vars — fallback used at first boot when meta has no entry. Useful for IaC-pinned deployments.

Backends: - local — self-contained. Each KMS key's master material is generated on the gateway, wrapped by the service master key, and persisted via meta. Admin endpoints under /_admin/kms/keys manage the key lifecycle. Most appropriate for on-prem deployments that don't have an external KMS. - vault — HashiCorp Vault's Transit secrets engine. Needs addr + token + transit_path (defaults to transit). The gateway calls transit/datakey/plaintext/{key} to generate per-object DEKs and transit/decrypt/{key} on GET. Per-object encryption context is canonicalized and base64-passed as Vault's context so decrypt fails on context mismatch (AEAD binding at the Vault layer). On external backends the /_admin/kms/keys endpoints return NotImplemented — manage keys directly in Vault. - disabled — SSE-KMS off; SSE-S3 still works if a service master key is set.

Runtime config API (all under /_admin/kms/*; gated by kms:Status/CreateKey/DeleteKey/KeyStatus): - GET /_admin/kms/config — current persisted backend (Vault token redacted; only token_set: true exposed). - PUT /_admin/kms/config — persist + hot-swap. Body is {backend, addr?, transit_path?, token?}. An empty token on update keeps the previously-stored one so admins can edit addr/path without re-entering secrets. - DELETE /_admin/kms/config — remove the persisted entry; next boot falls back to CLI/env. - POST /_admin/kms/test — run a throwaway generate_data_key + decrypt round-trip against the active backend. Returns {ok: bool, error: string|null} so the console can render a green/red indicator on the Save form.

Console: top-level Encryption page (admin-only) shows master-key status (read-only), backend selector (Disabled / Local / Vault), Vault addr + transit path + token-paste fields (token shown as •••••••• when already stored), Save & Apply button, and a Test-connection form.

Safety note: switching backends does not retroactively re-encrypt existing objects. Objects wrapped by the old backend stay readable only while that backend is still reachable; use CopyObject (phase 2f) to migrate them.

CopyObject with encryption transitions

  • Fast path (no data movement): when source and destination share the same SSE algorithm (and same KMS key id for SSE-KMS), CopyObject does a metadata-only copy — shards are shared between source and dest until either is deleted.
  • Slow path (decrypt → re-encrypt): when SSE settings differ (plaintext → SSE-S3, SSE-S3 → SSE-KMS, SSE-KMS → plaintext, SSE-KMS with different key, or explicit override via request headers), the gateway reads the source through its normal decrypt path, then re-PUTs through the normal encrypt path so the destination bucket's default — or the copy request's x-amz-server-side-encryption* headers — controls the new encryption.
  • SSE-C on copy: not yet supported; returns NotImplemented 501 when either side is SSE-C (needs the AWS-specific x-amz-copy-source-server-side-encryption-customer-* header set).

A re-encrypting CopyObject takes the source as a PUT takes its body: a source up to 16 MiB is held in memory, reserved from the gateway's memory budget; a larger one, up to S3's 5 GiB, is streamed from the source's stripes to the copy's, never held whole. A source over 5 GiB gets 400 InvalidRequest, as S3 answers; with no room in the budget, 503 SlowDown.

2. Data Protection (Erasure Coding)

Reed-Solomon (MDS)

  • Standard Maximum Distance Separable codes
  • Configurable k (data) + m (parity) shards
  • Any k shards can reconstruct the data
  • Presets: 3+2, 4+2, 8+4

LRC (Locally Repairable Codes)

  • Local parity groups for faster single-shard recovery
  • Three parameters: k (data) + local parity + global parity
  • Presets: 6+2+1, 10+2+2

Replication

  • Simple N-way replication (no erasure coding)
  • Configurable replica count (2-way, 3-way)

Bucket replication

  • Asynchronous replication of every new version to a bucket on another IOLake cluster, configured by the bucket's owner with S3's PutBucketReplication, naming a target (/_admin/replication/targets) of the bucket's own tenant, which that tenant's admin manages.
  • Replicas keep the source's version id and ETag; delete markers replicate if a rule asks; version deletes never do; replicas aren't replicated on.
  • Reliable by construction: the replication mark is written with the version, a scanner finds anything the immediate send missed, and the target stores a repeated send once.
  • Existing objects reach a target through a resync (/_admin/replication/resync, iolakectl replication resync), which survives restarts and shows its progress.
  • A per-target bandwidth limit, the cluster's, shared by every gateway's sends.
  • x-amz-replication-status, backlog and lag metrics. Guide: operations-guide/administration/replication.md.

Small-object packing

  • Small objects (above the 4 KiB inline cut-off, up to 64 KiB) are moved into packs: one erasure-coded stripe of about 1 MiB holding many objects. An 8 KiB object then takes about 1.6× its size on disk instead of 3× at 4+2.
  • Off by default. --pack-interval-secs N turns the packer on (aio and gateway); --pack-min-age-secs (default 3600) leaves recently written objects alone. One gateway packs at a time (a lease in meta).
  • Nothing changes for clients: an object reads, ranges, copies, locks and deletes the same packed or not. A pack is freed when its last object goes.
  • Crash-safe: a pack is recorded before it's written and sealed before any object points at it, and each pass reconciles what a packer that died left half done. Repair and drain handle packs from their record.
  • Compaction: a pack less than half live (its objects mostly deleted or overwritten) has its survivors moved into a new pack, and its space comes back. Packs shared by a copy are left alone.
  • Metrics: iolake_packs_written_total, iolake_pack_objects_total, iolake_pack_bytes_saved_total, iolake_pack_reconciled_total{action}, iolake_pack_compactions_total, iolake_pack_compacted_objects_total.

Storage Pools

  • Independent erasure coding and placement per pool
  • Failure domain configuration: OSD, node, rack, datacenter
  • OSD tags for hardware-class targeting (e.g., nvme, ssd)
  • Per-pool storage quotas
  • Protection presets with automatic feasibility check (greyed out if insufficient nodes)
  • Console shows efficiency %, min nodes, fault tolerance for each configuration
  • Buckets in pools: a bucket is created in a pool (x-iolake-pool on S3 CreateBucket, pool on the admin API, or the tenant's default_pool); tenants are limited to their allowed_pools; a pool in use can't be deleted. Pools currently select erasure coding and placement groups, not specific disks (OSD tags are planned).

2a. Placement Engine (CRUSH + Placement Groups)

Two-hop placement: object_id → pg_id → [osd_1..k+m]. Replaces per-object CRUSH with a fixed set of placement groups (PGs), each PG mapped to a specific (k+m)-tuple of OSDs. Modelled on Ceph's PG model + Cidon/Stutsman copyset diversity.

Every pool has placement groups

Every pool, the default one too, pre-allocates its PGs when it is made. Writes hash object key → PG via jump-consistent hash; PG → OSDs is an explicit Meta lookup, committed with an epoch before any write uses it.

Object → PG routing

  • Algorithm: Lamping & Veach jump consistent hash (Google 2014). O(log N) multiplications per call (~100 ns).
  • Minimal remap property: growing pg_count by one only moves old/new of keys, and every moving key lands on the new PG. Mostly academic today — resize isn't live — but the property is preserved for future PG-split support.
  • Key = xxh64(bucket/key).

PG → OSD assignment (copyset allocator)

  • Cidon/Stutsman copyset pattern: precompute a pool of valid (k+m)-tuples of OSDs that respect the pool's failure-domain level. PGs are sampled from this pool — dramatically reduces the number of distinct copysets and the correlated-data-loss probability when multiple OSDs fail simultaneously.
  • Failure-domain aware: every copyset spans distinct FD values (host, rack, datacenter, zone, region, etc. as configured on the pool).
  • Scatter width knob (balancer/scatter_width) — Cidon/Stutsman recommend ~10. Lower = fewer distinct copysets, lower loss probability, fewer choices for the balancer. Default 10.
  • Deterministic: seeded by topology version × pool hash so the copyset pool is reproducible across Meta replicas and restarts.

Placement Groups (PlacementGroup proto)

  • One Raft-replicated row per PG: (pool, pg_id, osd_ids, version, updated_at, migrating_to_osd_ids).
  • CAS-versioned — the balancer bumps version on every move.
  • Cached in-memory on every Meta replica via the apply-listener; hot reads don't hit redb.
  • Listed via GET /_admin/pools/{name}/placement-groups (paginated).

Balancer (descheduler pattern)

Leader-only background task. Evaluates every PG on each tick and commits moves via MultiCas when a PG is overloaded or underloaded.

Hysteresis gates (prevents thrash):

  • Overload: trigger a move when any OSD in the PG holds ≥ overload_multiplier × target PGs. Default 1.20 (20% above fair share).
  • Underload (optional): also trigger when any OSD in the cluster holds ≤ underload_multiplier × target PGs. Default 0.70 (30% below fair share); set to 0 to disable.
  • Improvement factor: candidate copyset's total load must be ≤ improvement_factor × current. Default 0.90 (≥10% improvement required to commit).

Concurrency cap: max(3, osds/3) moves per pool per tick by default. Bounds background churn. Override via balancer/per_tick_cap.

Greenfield mode: IOLake today has no live data migration. Balancer commits osd_ids directly — orphan shards on OSDs that dropped out of a PG are collected by the terminal-loss GC in the drain observer. If production data lands, the migrating_to_osd_ids field on PlacementGroup becomes the migration signal and the migrator task (not yet implemented) consumes it before committing the transition.

Hot-reloadable tuning (via /_admin/config/balancer/*)

Key Default Effect
balancer/sweep_interval_seconds 60 Tick cadence (clamped 5–3600)
balancer/overload_multiplier 1.20 Overload trigger threshold
balancer/underload_multiplier 0.0 Underload trigger (0 = off)
balancer/improvement_factor 0.90 Min improvement to commit
balancer/scatter_width 10 Copyset-pool diversity
balancer/per_tick_cap 0 (auto) Moves per pool per tick
balancer/paused false Halt all commits

Each knob re-reads on every tick — no restart required. Ticks themselves are skipped when the sweep interval changes so the clock stays monotonic.

Sizing

A pool made without a pg_count (the default pool too) gets one sized from the OSDs in service when it is made:

pg_count = 100 × OSDs in service / copies per PG,
           rounded up to a power of 2, from 256 to 4096

The 100 is config pg/members_per_osd. The reason is rebuild speed: a lost drive's PGs are rebuilt onto as many drives as it held PG members, so with too few PGs a large cluster rebuilds barely faster than a small one.

Cluster shape pg_count made
9 OSDs, 2+1 EC 512
24 OSDs, 4+2 EC 512
100 OSDs, 4+2 EC 2048
1,000 OSDs, 8+4 EC 4096 (the ceiling; give 8192 explicitly)

A count given when the pool is made is kept. Cost: ~200 bytes per PG row; pg_count=4096 is ~1 MB of Meta state per pool, and each PG is peered, scrubbed and recovered on its own. Hard cap is 65,536 per pool.

Scaling without PG split

pg_count is fixed at pool creation (v1 non-goal: growable pg_count). Three escape hatches cover the scale range without bolting on split:

  1. Size for target. Pick pg_count for the cluster you expect to reach, not the one you start at — cost is trivial.
  2. New pool per expansion (MinIO "server pool" pattern). When a pool's PG count becomes tight, create a new pool with more PGs and point new bucket creations at it. Existing buckets keep working on the old pool untouched. Zero-downtime because pool selection is per-bucket, recorded in BucketMeta at create time.
  3. Multiple pools per tier from day 1. nvme-1, nvme-2, … Buckets round-robin or least-full across siblings. Failure isolation + admin-op blast-radius containment come for free.

Iceberg workloads effectively never need PG split — the catalog indirection makes "move data to a new pool" a metadata-only flip rather than a full copy. Long-lived bucket-scoped workloads where the bucket name is the permanent client API are the only case where split is cheaper than migration; not a v1 concern.

Console visualization

/cluster/pools/{name} drill-down shows: - Stat cards (pg_count, k+m, target PGs/OSD, active OSDs). - Per-OSD PG-membership bar chart — red/amber coloring for overloaded/underloaded OSDs, target marker line. - PG × OSD matrix — scrollable grid, blue cell = OSD is a member, amber = migrating-to. Click a column to filter to one OSD's PGs; click a row to pin a single PG.

All backed by GET /_admin/pools/{name}/placement-groups + GET /_admin/nodes — no new data plane, just a read view.

Live status

GET /_admin/rebalance-status returns a unified view (legacy drift-rebalance fields + new PG counters): - pgs_moved_total — cumulative PG moves since process start - pg_candidates_last_tick — PGs flagged for a move in the last tick - pgs_scanned_last_tick — total PGs evaluated per tick - paused — merged signal of rebalance/paused OR balancer/paused

The console's Topology banner and the new Cluster → Balancing tab both consume this.

3. Data Lakehouse (Iceberg Tables + Delta Sharing)

Create analytics warehouses, manage Iceberg tables, query with Spark/Trino, and share data across organizations via the Delta Sharing protocol.

The Iceberg Tables and Table Sharing sections below describe the full data lakehouse capabilities.

4. Multi-Tenancy

Tenant Management

  • Create/List/Update/Delete tenants via /_admin/tenants API and console
  • Per-tenant configuration: Default pool, allowed pools, OIDC provider, admin users
  • Tenant quotas: Storage bytes, max buckets, max objects
  • Tenant labels: Arbitrary key-value metadata

Tenant-Scoped Resources

  • Users: Created with tenant field, ARN includes tenant (arn:objectio:iam::{tenant}:user/{name})
  • Access keys: Inherit tenant from user
  • Buckets: Created with tenant scope, ListBuckets filtered by tenant
  • Warehouses: Tenant-scoped Iceberg warehouses
  • Shares: Tenant-scoped table shares

Tenant Isolation

  • Auth middleware: AuthResult carries tenant from access key → user → tenant chain
  • Console session: Session token includes the user's tenant and ARN, and whether the sign-in made it a system administrator's; admin pages hidden for tenant users
  • API filtering: All list operations filter by authenticated user's tenant
  • System admin: the built-in admin user, and console sign-ins through the operator's identity provider, see all resources across tenants. Other system-scope users are not administrators

Tenant Admin Role

  • TenantConfig.admin_users: List of user IDs (or ARNs) who can admin the tenant without being system admins
  • POST /_admin/tenants/{name}/admins — grant tenant-admin
  • DELETE /_admin/tenants/{name}/admins/{user} — revoke
  • Three-tier access model: System admin (the built-in admin, or a sign-in through the operator's provider), tenant admin (in admin_users for their tenant), regular user
  • Scope: tenant admins can create/delete users + access keys + buckets + warehouses + shares within their own tenant; they cannot touch pools, tenants, nodes, system config, KMS backend, or other tenants
  • Console: dedicated "Manage admins" panel on each tenant row; picker lists that tenant's users + free-form user-id/ARN input
  • CLI: iolakectl tenant admin add|remove|list <tenant> <user> — see CLI

5. IAM (Identity & Access Management)

Full reference: Identity and Access and Public Access.

Users & Access Keys

  • Create/List/Get/Update/Delete users with display name, email, tenant
  • Suspend a user (PUT /_admin/users/{id} {"status": "suspended"}) or deactivate a key (PUT /_admin/access-keys/{id} {"status": "inactive"}) — refused within 15 s on every gateway, nothing deleted
  • Scoped keys: confined to s3://bucket/prefix/, read-only or read-write; refused on the admin API
  • Key format: AKIA* (permanent) or ASIA* (temporary)
  • Admin user: Auto-created on first meta service startup with credentials logged

Authentication

  • AWS SigV4 for requests and presigned URLs. The admin, Iceberg and Unity APIs also take a request signed over its body's hash without x-amz-content-sha256, as botocore's generic SigV4Auth and PyIceberg sign it; the body, up to 2 MiB, is hashed to check it. SigV2 is refused with 400 InvalidRequest, as S3 refuses it on buckets created since June 2020; with boto3 use botocore.config.Config(signature_version="s3v4").
  • Anonymous requests: served only what a bucket policy grants to everyone, with public access blocked by default
  • OIDC: the operator's provider and each tenant's own (Keycloak, Entra ID, Okta, Google, any compliant issuer); groups from any claim, dotted paths included; a console sign-in is the user bound to the token's issuer and subject, in the tenant its provider serves
  • STS AssumeRoleWithWebIdentity: an OIDC token for a role's temporary keys, AWS-wire-compatible; a tenant's roles trust only that tenant's provider, system roles only the operator's
  • Vended credentials for Iceberg / Unity clients, confined to a table's location and held to its warehouse's or catalog's tenant
  • Console session cookies with 24-hour TTL
  • Sessions, temporary credentials and listing tokens are signed with keys derived from a cluster secret held in meta's Raft log

Authorization

  • Tenant boundary: checked before any policy; no policy can grant across it
  • Bucket policies (PUT /{bucket}?policy) and IAM policies attached to users, groups and roles — per tenant, with a shared system catalogue (readonly, readwrite, writeonly)
  • Groups with members; roles with trust policies
  • Evaluation: explicit Deny wins; then any Allow; then the bucket's owner; else deny. No policy means owner-only.
  • Conditions: every IAM operator with IfExists, ForAnyValue:/ForAllValues: and Null; unknown operators refused at save. Keys include aws:SourceIp, aws:SourceVpce (named gateway endpoint), aws:SecureTransport, aws:username, s3:prefix, s3:ExistingObjectTag/*, the SSE request headers, obio:CredentialType
  • AWS policies work as written: arn:aws: ARNs and {"AWS": …} principals; NotPrincipal on Deny
  • Block Public Access at bucket, tenant and cluster level; new buckets start blocked
  • ACLs: bucket owner enforced — they grant nothing
  • Action namespaces: s3:*, iceberg:*, kms:*, sts:AssumeRoleWithWebIdentity
  • Admin access: the built-in admin user bypasses policy evaluation; a console session is an administrator's only when it was opened with the built-in admin's key or through the operator's identity provider.

Management SDKs

  • Go (github.com/io-lake/iolake-go-sdk) and Python (iolake-sdk) clients for the whole admin API, standard library only — see SDKs

3a. Iceberg Tables (REST Catalog)

Warehouse Management

  • POST/GET/DELETE /_admin/warehouses — Create, list, delete warehouses (the system admin, or an admin of the warehouse's tenant)
  • Backing bucket: Each warehouse automatically creates an S3 bucket (iceberg-{name}); a warehouse is deleted only once it is empty
  • Tenant-scoped: A caller sees and uses only its own tenant's warehouses
  • Warehouse-scoped namespaces: Each warehouse has its own namespace/table hierarchy, policies included

Iceberg REST Catalog API

  • GET /iceberg/v1/config — Catalog configuration (warehouse location)
  • Namespace CRUD: Create, load, drop, list, update properties, check existence
  • Table CRUD: Create, load, commit (CAS), drop, list, rename, check existence
  • Multi-level namespaces: Dot-separated hierarchy (e.g., db.schema)
  • Namespace policies: IAM-style access control per namespace
  • Table policies: Per-table access control
  • Evaluation: as for a bucket policy: explicit Deny wins, then any Allow, then the warehouse's administrators; else deny. No policy means admins only.
  • Table locations: confined to the warehouse's bucket
  • Data filters: Column/row-level security per principal

Authentication

  • SigV4: Same access key / secret key as S3 (industry standard)
  • OIDC/OAuth2: Bearer token authentication via external provider
  • Console session cookie: For web console access
  • Unified auth middleware: Tries SigV4 → OIDC → session cookie; a missing or failing credential is refused with 401 (open only under --no-auth)

Vended Credentials (STS)

  • LoadTable response: Includes temporary S3 credentials in config field, read-write for a caller allowed iceberg:UpdateTable and read-only otherwise; a caller not allowed iceberg:LoadTable gets 403 and none
  • Credential format: ASIA* access key + secret + HMAC-signed session token
  • 1-hour TTL: Fresh credentials generated per LoadTable call
  • Path-style access: s3.path-style-access=true for IOLake compatibility
  • External endpoint: Configured via --external-endpoint for public access

Client Compatibility

  • Catalog URI http://<gateway>/iceberg (clients add /v1), warehouse = a warehouse's name
  • PyIceberg: SigV4 signing (rest.sigv4-enabled=true), over each body's hash without x-amz-content-sha256 (botocore's generic signer), which the gateway hashes the body to check; the code repo's examples/pyiceberg runs against iolake-aio
  • Spark, Trino: the REST catalog clients, with SigV4 or OAuth2
  • No query engine is tested against the catalog in CI yet (iolake#8)

3b. Unity Catalog (REST API)

Databricks-shape governance plane for catalogs / schemas / tables / functions / volumes / models / model-versions, mounted at /api/2.1/unity-catalog/*. Coexists with the Iceberg REST Catalog — they share the same auth bridge, the same BucketPolicy/PolicyEvaluator backbone, and the same admin gate.

Three-level entity model

  • Catalog — top of the namespace; auto-provisions an unity-{name} backing bucket; belongs to its creator's tenant (created by a catalog admin or a tenant admin), and only that tenant's callers see it
  • Schema — addressed as {catalog}.{schema}
  • Table / Function / Volume / Model — addressed as {catalog}.{schema}.{name}; an EXTERNAL table's or volume's storage_location must lie inside the catalog's bucket (s3://unity-{catalog}/...), where its vended credentials are scoped
  • Model version — addressed as {catalog}.{schema}.{model}/versions/{n}

Policy model

  • IAM-style JSON — same BucketPolicy shape as S3 / Iceberg; unity: action prefix and arn:obio:unity:::{catalog}[/{schema}[/{table}]] resource ARNs
  • Three-level walk — catalog → schema → table, with the caller's attached IAM policies; deny at any level wins, then any Allow, then the catalog's administrators (catalog admins and its tenant's admins); else deny
  • Grantable actions — unity:Get*, unity:Create*, unity:Delete* for every entity, plus unity:SetTableSecurity for row-filter / column-mask binding (grantable to non-admin via Unity policy — does not require admin)
  • Per-entity policy editor — PUT /api/2.1/unity-catalog/{catalogs|schemas|tables}/{name}/policy, GET to read, empty body clears

Row filters & column masks

  • PUT /api/2.1/unity-catalog/tables/{full_name}/security — binds a SQL UDF as a row filter and/or per-column masks
  • Validation — row-filter UDF must return BOOLEAN; mask functions must exist in the same catalog.schema; rejects unknown function references at config time
  • Engine-side enforcement — engines (Spark/Trino) read the binding from GET /tables/{full_name} and inject WHERE function(input_cols) / mask_function(col, …) at query compile time. Unity itself does not execute UDFs

UDF context built-ins

  • GET /api/2.1/unity-catalog/me returns {user_name, user_id, user_arn, groups, group_arns, tenant, is_admin} — the canonical principal context engines use to populate the SQL built-ins that row-filter and column-mask UDFs reference:
Built-in Source field
current_user() user_name
is_member(g) groups contains g
is_account_group_member(g) groups contains g

Default deny direct-S3 on backing buckets

  • Auto-attached at catalog create — bucket policy on unity-{catalog} denies any access where obio:CredentialType != STS
  • Forces vended-credential flow — permanent IAM keys can't bypass row filters / column masks by going around Unity to the backing bucket
  • obio:CredentialType policy variable — surfaced from AuthMode (set to STS in the STS auth path, Permanent in SigV4/OIDC, Anonymous in --no-auth); usable in any bucket policy via StringNotEquals / StringEquals conditions

Vended credentials (STS)

  • POST /…/temporary-table-credentials — mint scoped 1-hour STS session for table data access
  • POST /…/temporary-volume-credentials — same for governed volumes
  • Policy-gated (unity:GenerateTemporaryCredentials) — recipients of the vended creds appear to the gateway as obio:CredentialType: STS and pass the default-deny policy; the session is held to the catalog's tenant

OIDC group bridging

  • Name-based mapping — Keycloak / OIDC group claim "data-team" maps to local IAM group "data-team" by exact name match
  • Dual stitching — every authenticated request gets BOTH the IAM group ARN (for policies that name the IAM ARN directly) AND the IAM group_id (so list_attached_policies(group_id=…) returns the right set), so a single IAM policy attached to the IAM group governs SigV4 and OIDC users uniformly

Console UI

  • Unity Catalog page — three-pane catalog → schema → entity navigation; Tables / Functions / Volumes / Models tabs per schema; full CRUD modals
  • Shield button — opens the PolicyEditor modal scoped to the entity
  • Filter button — opens the TableSecurityEditor modal (function dropdown filtered to same-schema UDFs, multi-mask rows); the icon turns green when a row filter or column mask is active on the row
  • Policy templates — Policies page generator now ships Unity-shape templates (unity-readonly, unity-data-engineer, unity-ml-engineer, unity-security-officer, unity-deny-delete) and an S3-shape s3-deny-non-sts template that mirrors the Unity backing-bucket default

Client compatibility

  • Databricks SDK — Unity REST API is wire-compatible with Databricks Unity Catalog (/api/2.1/unity-catalog/* paths, snake_case JSON)
  • PySpark with Unity connector — bind warehouse via standard config; SigV4 or OIDC for auth
  • Trino Unity connector — same

3c. Table Sharing (Delta Sharing Protocol)

Share Management

  • Create/List/Delete shares via admin API and console: GET/POST /_admin/shares, DELETE /_admin/shares/{share}, GET/POST /_admin/shares/{share}/tables, DELETE /_admin/shares/{share}/schemas/{schema}/tables/{table}
  • Tenant-scoped shares: Filtered by authenticated user's tenant; managed by the system admin or the share tenant's admins
  • A share exposes its own tenant's data: a table in another tenant's bucket is added only by the system admin

Table Types

  • Delta Lake: Share Delta tables by S3 bucket + path (table_type: "delta")
  • Iceberg tables are refused (table_type iceberg, uniform or none): adding one answers 400, and the version, metadata and query calls refuse one already in a share. A share serves the files of a Delta log, which an Iceberg table doesn't have. Read Iceberg tables through the Iceberg REST catalog.

Recipient & Token Management

  • Create recipients: Name + list of accessible shares (POST /_admin/recipients, DELETE /_admin/recipients/{name}), by an admin of every share listed
  • Bearer tokens: One-time display, SHA-256 hashed for storage
  • profile.share: Standard Delta Sharing credential file format
  • Token authentication: Recipients authenticate via Authorization: Bearer <token>

Protocol Endpoints

  • GET /delta-sharing/v1/shares — List accessible shares
  • GET /delta-sharing/v1/shares/{share}/schemas — List schemas in share
  • GET /delta-sharing/v1/shares/{share}/schemas/{schema}/tables — List tables
  • GET .../tables/{table}/version — Current table version (snapshot ID)
  • GET .../tables/{table}/metadata — Table metadata (NDJSON format)
  • POST .../tables/{table}/query — Query table data (returns presigned S3 URLs)

Data Access

  • Presigned URLs: Server generates SigV4 presigned URLs for Parquet data files
  • 1-hour expiry: Presigned URLs valid for 1 hour
  • No S3 credentials needed: Recipients access data via presigned URLs only

6. File Storage (NFS / POSIX)

File protocols (NFS, SMB) are not built yet; they are roadmap track E. What exists today is file-like access through S3:

Current File-Like Access (via S3)

  • Object Browser (Console): Navigate bucket → folder → file hierarchy
  • Presigned URLs: Time-limited download/upload URLs for direct file access
  • Folder semantics: Delimiter-based folder listing (?delimiter=/)
  • Range reads: HTTP Range header for partial file reads
  • Multipart upload: Large file upload in parts (up to 10,000 parts)

7. Block Storage

gRPC Block Service

  • Thin provisioning: Storage allocated only on write
  • 4 MB chunks: Logical block addresses mapped to erasure-coded chunks on OSDs
  • 512-byte LBA: Standard sector size
  • Snapshots: Copy-on-write snapshots and writable clones
  • Write journal: Crash-consistent writes via write-ahead journal
  • Write cache: Coalesces small writes before flushing to chunks

QoS (Quality of Service)

  • Not built. UpdateVolumeQos, and a CreateVolume that carries QoS settings, answer UNIMPLEMENTED rather than accept limits nothing enforces; so do GetVolumeStats and the OSD, cluster and I/O-trace calls (the block gateway's /metrics has its I/O).

Protocol Support

  • gRPC BlockService: Native block I/O over gRPC (port 9300)
  • NBD (Network Block Device): Linux kernel NBD attachment (port 10809)

Block Gateway

  • Stateless, horizontally scalable
  • Accepts block I/O, buffers in write cache, erasure-encodes 4 MB chunks, flushes to OSDs
  • Bridges block storage volumes to the erasure-coded object layer

8. Web Console

Signing in

  • Sign-in page: access key and secret key, or single sign-on through the operator's or a tenant's OIDC provider
  • Session: HMAC-signed HttpOnly cookie, 24-hour TTL; signing out clears it
  • Two consoles: the ops console (system administrators, every page) and the tenant console (a tenant's own buckets, tables, users and policies); each can be served on its own listener

The frame

  • Navigation grouped by job: Overview · Storage (Buckets) · Lakehouse (Catalog, Sharing) · Cluster (Health, Nodes and drives, Pools and placement, Performance) · Access (Tenants, Users and keys, Policies, Sign-in providers) · Security (Encryption). The tenant console shows only its own pages.
  • Top bar on every page: the breadcrumb, and the cluster's data protection read from every placement group's state: all data protected, rebuilding, short of a drive, or at risk. It links to Cluster health.
  • Jump to a page: ⌘K or Ctrl+K searches the pages by name and keyword.
  • Themes: light and dark, following the operating system unless pinned from the sidebar.

Pages

Page What it shows
Overview Data protection first (clean, rebuilding with what is left, or at risk), then capacity, S3 requests, p99 latency, objects and drives; throughput and latency charts (from Prometheus when configured, else since the page opened); a "Needs attention" list from the cluster's state (groups not clean, drives down, out, draining, unhealthy or near full, rebalancing paused, server errors), each with its fix; capacity by host or drive.
Buckets Filterable table (tenant, pool, versioned, suspended, locked, public) with stored bytes and objects; a details panel for the selected bucket: protection and the pool's health, its settings in plain words, who has access. Create bucket.
Bucket (/buckets/:name) Tabs: Objects (the object browser: folders, search, upload, new folder, bulk download and delete, an object's details), Settings (versioning, object lock, protection, encryption, replication, delete), Access (owner, public access, bucket policy), Lifecycle (rules). Old /objects links open the bucket and folder they named.
Catalog Iceberg warehouses and Unity catalogs in one tree. A table shows its snapshots, schema, partitioning, properties, where it lives and who can reach it, with connection snippets for PyIceberg, Spark and Trino; create and delete warehouses, namespaces, catalogs and schemas. Unity's functions, volumes and models are managed in its advanced view, linked from the catalog.
Sharing Delta Sharing shares, their tables and recipients; tokens and profile.share files.
Cluster health Placement groups by state and pool; what recovery is doing and why it is stuck; drains, rebalancing and scrubs; every drive by rack and host, with drain, mark out and mark in; hosts.
Nodes and drives Hosts, OSDs and their disks, with tabs for Topology (region → zone → datacenter → rack → host → OSD) and Balancing (status, pause and resume, tuning). Add a host through the host provider.
Pools and placement Pools with their protection (RS, LRC, replication), failure domain and placement-group count; a pool's placement (/cluster/pools/:name): groups by state, per-OSD membership, scrub a group.
Performance S3, Iceberg and request metrics as charts, from Prometheus or the live scrape.
Tenants Tenants with their quotas, pools, sign-in provider and admins.
Users and keys Users, service accounts, groups and access keys.
Policies Named IAM policies, attached to users or groups.
Sign-in providers OIDC providers (Entra ID, Keycloak, Okta, Google, any compliant issuer).
Encryption SSE master-key status and the KMS backend (disabled, local, Vault), with a test.

Design

  • Sora for titles and figures, IBM Plex Sans for text (13px base), IBM Plex Mono for identifiers and table headers
  • One page layout: title, one line saying what the page holds, actions on the right, then content; details open in a side panel
  • Status always carries a word as well as a colour; text meets 4.5:1 contrast in both themes; form fields are labelled for screen readers

9. Infrastructure

Architecture

  • Gateway (stateless, horizontally scalable): S3 API, Iceberg REST Catalog, Delta Sharing, Admin API, Console
  • Meta (openraft 0.9 cluster): Bucket/object metadata, placement (CRUSH 2.0), IAM, tenant config. One code path; deployment picks single-voter (replicas=1) or HA quorum (replicas=3+).
  • OSD (one per disk): Erasure-coded shard storage with Direct I/O, WAL
  • Block Gateway (optional): Block storage over gRPC + NBD

Meta HA (openraft 0.9)

  • Real consensus, not a placeholder: RaftStorage implemented over redb (log + vote + state in raft_logs, raft_vote, raft_state tables); RaftNetwork over tonic gRPC on the existing meta port, JSON-encoded envelopes
  • Cluster size is a deployment choice: meta.replicas=1 boots a single-voter cluster (no election drama, same operational behavior as pre-Raft meta); meta.replicas=3+ gets true quorum, leader election, log replication. Same binary, same code path
  • Bootstrap automated via helm: a Job created with each install and upgrade calls /init on meta-0 unless the cluster exists, adds the pods that aren't voters as learners through the leader, and promotes the voter set; with every pod a voter it changes nothing
  • Admin API at :9102 per meta pod:
  • POST /init — bootstrap a single-voter cluster (idempotent)
  • POST /add-learner {node_id, addr} — onboard a peer non-voting
  • POST /change-membership {voters} — promote the voter set
  • GET /status — leader id, term, voters, learners, last_log_index, last_applied
  • GET /health — 200 while this node's Raft runs, 503 once it has stopped (the chart's liveness probe)
  • GET /ready — 200 while this node is a member with a leader known, 503 with the reason otherwise (the chart's readiness probe)
  • Every meta write goes through Raft: config, buckets, IAM, tenants, OSD registrations, KMS keys, multipart uploads; replicas apply them in order
  • Any meta node serves any client: a follower forwards each gRPC call to the leader (x-objectio-forwarded); clients take a comma-separated --meta-endpoint list, balanced over health-probed nodes, with a 5 s call timeout, so a dead or frozen meta is routed around
  • Snapshots: the whole state machine, built from the tables on demand and installed in full on a lagging or new node
  • Tested: multi-process e2e (tests/e2e/tests/meta_ha.rs) kills, freezes and replaces metas under S3 traffic

Deployment

  • Kubernetes (Helm): Production-grade Helm chart with StatefulSets, PVCs, Ingress
  • Docker Compose: Local development cluster (3 meta + 6 OSD + 1 gateway)
  • Single binary: All-in-one Docker image with per-service entrypoints

Observability

  • Prometheus metrics: Exported at /metrics on each service
  • Per-operation counters: iolake_s3_requests_total, iolake_iceberg_requests_total
  • Per-operation latency: Duration histograms
  • OSD status: Disk capacity, shard counts, SMART health
  • Node info: K8s node name (downward API), OS version, CPU, RAM, IOLake version

Networking

Service API Port Metrics Port
Gateway 9000 9000 /metrics
Meta 9100 9101
OSD 9200 9201
Block Gateway 9300 (gRPC) / 10809 (NBD) —

Storage Engine

  • 4 MiB chunk/stripe size: Fundamental EC unit
  • 4 KiB block size: Internal allocation unit; a shard takes a run of whole blocks, with a header and footer carrying its checksum
  • O_DIRECT / F_NOCACHE: Bypass OS page cache; io_uring on Linux release builds
  • Flush before acknowledge: a shard is fsynced, then its location, before the write returns
  • Metadata store: a write-ahead log in front of an on-disk index with a bounded cache (--meta-cache-mib), so an OSD's memory doesn't grow with its object count
  • Shard checksums: each shard's CRC32C is recorded with the object and checked on every read and repair; a background scrub reads every shard at a set rate
  • CRUSH 2.0 placement: Consistent hashing with failure domain awareness

Cluster Topology & Locality

  • 5-level failure domain model on OsdNode and FailureDomainInfo: region → zone → datacenter → rack → host. Additive proto change, back-compat with 3-level deployments (empty zone/host = "inherit from enclosing")
  • TopologyDistance metric (iolake-placement::distance): ranks two peers as SameHost < SameRack < SameDatacenter < SameZone < SameRegion < Remote < Unknown. Total order — safe to use as a sort key. Empty-on-both-sides at a level counts as matching so deployments that only use some levels aren't penalized
  • Placement feasibility validation: Crush2::validate_placement_feasibility(level, required_count) returns InsufficientDomainSpread { required_level, required_count, available_count } when a pool cannot spread across the configured level. Surfaced via GET /_admin/placement/validate?pool=NAME
  • Gateway self-positioning: --topology-region/zone/datacenter/rack/host CLI flags (or IOLAKE_TOPOLOGY_* env vars). Stored on AppState.self_topology; visible via GET /_admin/cluster-info
  • Locality-aware EC reads: get_object ranks all candidate shard positions by distance(gateway, shard_host), fetches the k closest first. Parity reconstruction only triggers when nearer shards are unhealthy. No protocol change — same ReadShard RPCs, different ordering
  • GET /_admin/topology — aggregated region→zone→datacenter→rack→host→OSDs tree + per-level distinct counts
  • GET /_admin/cluster-info — gateway's self-position + every OSD's computed distance + is_local flag
  • Prometheus metric: iolake_read_locality_bytes_total{locality="same-rack"|"same-zone"|"same-datacenter"|"same-region"|"remote"|"unknown"} — bytes fetched from OSDs partitioned by topology distance. Multi-DC deployments use this to prove cross-DC bandwidth is being collapsed
  • CLI: iolakectl cluster topology (full tree) and cluster validate-placement <pool> (exits non-zero when a pool is unsatisfiable)

Stable OSD identity

  • OSD registration retry: register_with_meta backs off exponentially (2s → 30s cap) until it succeeds. Stops OSDs from silently staying out of the cluster after a meta-first rolling restart
  • Address-based dedupe: re-registration at an existing address with a different node_id evicts the stale entry from both osd_nodes and the CRUSH topology. One-shot dedupe on meta startup also cleans ghosts left by older binaries