Skip to content

Configuration Reference

IOLake binaries are configured with CLI flags, some of which can also be set by an environment variable (shown as [env: …] in each binary's --help). Only iolake-osd reads a TOML file (--config); meta and the gateway take flags only. A flag wins over its environment variable, which wins over the OSD's file, which wins over the built-in default.

This page documents the flag surface for each binary against the current main; <binary> --help is the complete list. Only the env vars listed here are honoured.


Ports

Every binary's defaults are the cluster's ports, the ones the Helm chart and these guides use:

Role Port Protocol
gateway S3 9000 HTTP
meta gRPC (and Raft) 9100 gRPC
meta metrics 9101 HTTP
meta admin (Raft bootstrap / membership) 9102 HTTP; HTTPS under mTLS
OSD gRPC 9200 gRPC
OSD metrics 9201 HTTP
block-gateway gRPC 9300 gRPC
block-gateway NBD 10809 TCP (NBD)

The clients' defaults follow: the gateway's, the OSD's and the block gateway's --meta-endpoint default to http://localhost:9100. Beyond one host, pass every address. A host running several OSDs gives each its own ports: iolake-install puts OSD i on 9200 + 2i and its metrics on 9201 + 2i.

---|---|---|---| | gateway S3 | 9000 | 9000 | HTTP | | meta gRPC | 9100 | 9001 | gRPC | | meta metrics | 9101 | 9101 | HTTP | | meta admin (Raft bootstrap / membership) | 9102 | 9102 | HTTP; HTTPS under mTLS | | OSD gRPC | 9200 | none without a config file; 9002 when osd.toml has no listen | gRPC | | OSD metrics | 9201 | 9201 | HTTP | | block-gateway gRPC | 9300 | 9300 | gRPC | | block-gateway NBD | 10809 | 10809 | TCP (NBD) | | block-gateway metrics | 9301 | 9301 | HTTP |

Every port but the gateway's is internal: who may reach each is in the port table.

The chart's values are what you should document to end users. The code defaults only matter if you run the binaries standalone with no --listen override. The clients' defaults follow the code, not the convention: the gateway's --meta-endpoint defaults to http://localhost:9001, the block gateway's to http://localhost:9100. Pass every address explicitly.


iolake-gateway

S3 / Iceberg / Delta Sharing / admin / console.

Flags

iolake-gateway \
  --listen 0.0.0.0:9000 \
  --meta-endpoint http://meta1:9100,http://meta2:9100,http://meta3:9100 \
  --ec-k 4 --ec-m 2 \
  --region us-east-1
Flag Default Purpose
-l, --listen <addr> 0.0.0.0:9000 Data plane listen — S3 + Iceberg + Delta Sharing. In single-port mode (no split flags) also serves /_admin/*, /metrics and /_console/*.
--admin-listen <addr> empty Optional dedicated listener for /_admin/* + /metrics. Set this and admin moves off --listen entirely (see Listener split).
--ops-console-listen <addr> empty Optional dedicated listener for the ops console SPA + /_admin/* (system-admin surface).
--tenant-console-listen <addr> empty Optional dedicated listener for the tenant console SPA + /_admin/* (end-user surface).
--meta-endpoint <urls> http://localhost:9100 Meta nodes: one address, or every meta node's, comma-separated. Any node serves any call (a follower forwards it to the Raft leader), and one that stops answering is skipped. See Meta high availability.
--ec-k <n> 4 EC data shards
--ec-m <n> 2 EC parity shards
--protection <kind> ec ec, lrc, or replication
--lrc-local-parity <n> 0 LRC local parity count (with --protection=lrc)
--lrc-global-parity <n> 0 LRC global parity count
--replicas <n> 3 Replica count (with --protection=replication)
--no-auth off Disable SigV4 — dev only
--prometheus-url <url> empty A Prometheus that scrapes the cluster: the console's long ranges and iolakectl metrics query go through it. Without it /_admin/metrics/query answers that none is configured. Env IOLAKE_PROMETHEUS_URL.
--no-reexport-metrics off Leave the OSDs' and meta's metrics out of this gateway's /metrics; set it with more than one gateway (where to scrape). Env IOLAKE_NO_REEXPORT_METRICS.
--no-bucket-metrics off Leave per-bucket iolake_bucket_* series out of /metrics. Env IOLAKE_NO_BUCKET_METRICS.
--inline-max-size <bytes> 4096 Objects up to this size are stored in their metadata record; 0 turns it off
--small-shard-max <bytes> 16384 Shards up to this size travel with the object's metadata copies (at most 16384); 0 turns it off
--endpoint-name <name> empty Name of the --listen endpoint, seen by policies as aws:SourceVpce (see Public Access). Env IOLAKE_ENDPOINT_NAME.
--data-listen <addr>=<name> — A further data-plane listener (S3, Iceberg, Delta Sharing; no admin, no console) with its own endpoint name. Repeatable.
--audit-log <path> unset Append every request's audit event as JSON lines to this file (- for stdout). Env IOLAKE_AUDIT_LOG.
--audit-spool <dir>, --audit-system-bucket <name> unset The audit spool and the system's own audit bucket; the other --audit-* and --bucket-log-* flags are in audit and bucket logging
--lifecycle-interval-secs <n> 3600 How often the lifecycle worker scans buckets with rules
--pack-interval-secs <n>, --pack-min-age-secs <n> 0 (off), 3600 Small-object packing
--replication-scan-secs <n> 30 How often the replication scanner looks for unsent versions
--trusted-proxies <cidrs> empty Proxies whose X-Forwarded-For / X-Forwarded-Proto are believed for aws:SourceIp / aws:SecureTransport. Env IOLAKE_TRUSTED_PROXIES. Set it when TLS ends at an ingress.
--memory-budget <size> half the memory limit, else 512MiB Memory for request bodies and GET read-ahead (512MiB, 2GiB); a request that doesn't fit gets 503 SlowDown. Unset: half the process's cgroup memory limit when it has one. Env IOLAKE_MEMORY_BUDGET. See gateway memory.
--max-requests <n> 1024 Requests served at once, each until its response is sent; more get 503 SlowDown. /health, /_ready and /metrics aren't counted. Env IOLAKE_MAX_REQUESTS.
--header-timeout-secs <n> 30 How long a request's headers may take to arrive, and an idle keep-alive connection waits for its next request. Env IOLAKE_HEADER_TIMEOUT_SECS.
--body-timeout-secs <n> 30 How long a request body may send no byte (400 RequestTimeout), and a client may read no byte of a response (disconnected). Env IOLAKE_BODY_TIMEOUT_SECS.
--shutdown-delay-secs <n> 5 On SIGTERM, how long /_ready answers 503 while the gateway still serves, before it stops taking connections. Env IOLAKE_SHUTDOWN_DELAY_SECS. See how a node stops.
--shutdown-drain-secs <n> 20 On shutdown, how long requests in flight are given to finish once the gateway stops taking connections. Env IOLAKE_SHUTDOWN_DRAIN_SECS.
--region <name> us-east-1 The region requests are signed for (SigV4), and the one GetBucketLocation answers for every bucket
--external-endpoint <url> empty Public URL (Iceberg vended creds, Delta Sharing presign, OIDC callback)
--s3-domain <domain> none A domain clients address buckets under by host name, e.g. s3.example.com: photos.s3.example.com is bucket photos. Repeatable, or comma-separated. Env IOLAKE_S3_DOMAIN. Needs wildcard DNS and a wildcard certificate for the domain; see virtual-hosted addressing.
--master-key-env <name> IOLAKE_MASTER_KEY Env var name holding the base64 SSE master key
--kms-backend <kind> local local or vault
--oidc-issuer-url <url> unset Enables OIDC login for console + Iceberg JWTs
--oidc-client-id <id> unset OIDC client ID
--oidc-client-secret <val> unset OIDC client secret
--oidc-audience <val> client id JWT aud expected value
--oidc-groups-claim <name> groups Claim mapping to group/role set
--oidc-role-claim <name> role Claim mapping to single role
--oidc-scopes <scopes> openid profile email Space-separated scope list
--oidc-admin-roles <roles> empty Comma-separated list of admin-granting roles
--delta-access-key-id <ak> empty Presign AK for Delta Sharing recipient URLs
--delta-secret-key <sk> empty Presign SK for Delta Sharing recipient URLs
--delta-url-ttl-seconds <n> 0 (3600) Lifetime of Delta Sharing presigned file URLs
--host-provider <kind> noop noop (refuses host actions) or k8s — powers /_admin/hosts and /_admin/osds/*/reboot
--host-provider-namespace <ns> default Namespace for the k8s provider (from $POD_NAMESPACE)
--host-provider-osd-sts <name> iolake-osd OSD StatefulSet name for the k8s provider
--topology-region <val> empty Gateway's own topology — drives locality-aware reads
--topology-zone <val> empty
--topology-datacenter <val> empty
--topology-rack <val> empty
--topology-host <val> empty
--log-level <lvl> info trace, debug, info, warn, error; RUST_LOG, when set, wins
--tls-cert, --tls-key, --tls-ca unset mTLS to meta and the OSDs (every binary has these). Env IOLAKE_TLS_CERT, IOLAKE_TLS_KEY, IOLAKE_TLS_CA.

Environment variables

Besides those named in the table above, the clap parser reads these directly:

Env Flag
IOLAKE_TOPOLOGY_REGION --topology-region
IOLAKE_TOPOLOGY_ZONE --topology-zone
IOLAKE_TOPOLOGY_DATACENTER --topology-datacenter
IOLAKE_TOPOLOGY_RACK --topology-rack
IOLAKE_TOPOLOGY_HOST --topology-host
IOLAKE_HOST_PROVIDER --host-provider
IOLAKE_HOST_PROVIDER_OSD_STS --host-provider-osd-sts
POD_NAMESPACE --host-provider-namespace

Read separately from the flag surface:

Env Purpose
IOLAKE_MASTER_KEY 32-byte base64 SSE-S3 master key
IOLAKE_CONSOLE_DIR Directory holding both console bundles, ops/ and tenant/ (default /usr/share/objectio/console): the base for the two below.
IOLAKE_OPS_CONSOLE_DIR Path to the ops SPA bundle. Default: $IOLAKE_CONSOLE_DIR/ops. Served on --ops-console-listen.
IOLAKE_TENANT_CONSOLE_DIR Path to the tenant SPA bundle. Default: $IOLAKE_CONSOLE_DIR/tenant. Served on --tenant-console-listen.
RUST_LOG Override --log-level; tracing directive syntax

Listener split

By default the gateway is a single-port deployment: --listen carries S3, Iceberg, Delta Sharing, /_admin/*, /metrics, and /_console/* all on the same socket. This matches MinIO's default and is fine for small clusters.

For larger or regulated deployments you can split the four planes across separate listeners by setting any combination of --admin-listen, --ops-console-listen, --tenant-console-listen. Each binds independently; you can enable one, two, or all three.

Listener Example Routes mounted Audience
data --listen 0.0.0.0:9000 /, /{bucket}*, /iceberg/v1/*, /api/2.1/unity-catalog/*, /delta-sharing/v1/*, /health Public — SDK / CLI / Spark / Trino
admin API --admin-listen 127.0.0.1:9001 /_admin/*, /metrics, /health CLI, third-party integrations (Datadog, billing, OTel)
ops console --ops-console-listen 127.0.0.1:9002 /_console/* (ops SPA), /_console/api/*, /_admin/*, /health System operators on a mgmt VLAN
tenant console --tenant-console-listen 0.0.0.0:9003 /_console/* (tenant SPA), /_console/api/*, /_admin/*, /health End users — safe to expose publicly

Behavior in split mode:

  • /_admin/* is mounted on the admin API listener AND on each console listener (the browser stays same-origin — no CORS). It is removed from the data listener — /_admin/* requests there fall through the S3 wildcard and return SigV4 errors.
  • /metrics lives on the admin API listener only.
  • /_console/* is removed from the data listener; each console listener serves its own SPA bundle from IOLAKE_OPS_CONSOLE_DIR / IOLAKE_TENANT_CONSOLE_DIR.
  • The startup log emits a WARN if --admin-listen or --ops-console-listen is bound to 0.0.0.0 — the intent of the split is to firewall the control plane to a mgmt interface.
  • The same /_admin/* handlers run on every listener that mounts them. Per-tenant scope is enforced server-side (see require_tenant_admin_access), so the tenant console listener is not a privilege-elevation path even though it exposes the same surface as the ops console.

Why two consoles, not one with role-gating: see architecture/console-split.

Example — public S3 on the WAN, control plane on a mgmt VLAN:

iolake-gateway \
  --listen                  0.0.0.0:9000 \
  --admin-listen           10.0.5.10:9001 \
  --ops-console-listen     10.0.5.10:9002 \
  --tenant-console-listen   0.0.0.0:9003 \
  --meta-endpoint http://meta1:9100 \
  --region us-east-1

Omit all three split flags and the gateway serves everything on --listen.


iolake-meta

Raft metadata cluster.

Flags

iolake-meta \
  --node-id 1 \
  --listen 0.0.0.0:9100 \
  --raft-advertise meta1:9100 \
  --data-dir /var/lib/objectio/meta

A meta node starts outside any cluster. The Raft group is formed over the admin port: POST /init on the first node, then on the leader POST /add-learner for each other node ({"node_id": 2, "addr": "meta2:9100"}, the address being the peer's gRPC port, 9100) and POST /change-membership with every voter ({"voters": [1, 2, 3]}). A single node needs only /init. See Linux and meta HA.

Flag Default Purpose
--node-id <n> 1 Raft node ID; each node of a group needs its own
-l, --listen <addr> 0.0.0.0:9100 gRPC listen — MetadataService + Raft RPC
--ec-k <n> 4 Default EC data shards
--ec-m <n> 2 Default EC parity shards, at least 1. With --ec-k, a profile every loss of m shards decodes (which); meta refuses to start with another, 1+0 too
--replication <n> unset Use replication instead of EC: n copies of each object, from 1 (one copy, no redundancy: a one-OSD dev cluster)
--data-dir <path> /var/lib/objectio/meta Persistent Raft + redb state
--admin-user <name> admin Initial admin username (created on first startup)
--metrics-port <n> 9101 Prometheus scrape port (0 none)
--admin-port <n> 9102 Admin port — Raft bootstrap + membership (/init, /add-learner, /change-membership) and the node's state (/status, and /health and /ready for probes). HTTPS under mTLS, where every call but reading the state takes a client certificate of role admin, and reading it takes any client, with a certificate or none; plain and unauthenticated without
--admin-host <ip> the host of --listen Address the admin port binds: with --listen 0.0.0.0:… every interface, with loopback this host only
--raft-advertise <addr> empty Address advertised to Raft peers (defaults to --listen)
--repair-interval-secs <n> 3600 Seconds between repair passes (leader); 0 off
--drain-interval-secs <n>, --drain-batch <n> 30, 64 Drain sweeps, and shards moved per draining OSD per sweep
--min-free-mib <n> 1024 Free space meta keeps on the filesystem of --data-dir: below it the leader refuses writes (507) and a follower takes its leader's entries down to half of it; reads go on, and writes resume once there is room (meta's disk filling)
--raft-snapshot-every <n> 500000 Log entries between snapshots; the log is compacted into each
--raft-keep-logs <n> 10000 Entries kept behind the last snapshot
--raft-checkpoint-ms <n> 200 How often applied state is made durable
--raft-heartbeat-ms <n> 500 Raft heartbeat interval; also how long the leader waits for a follower to take entries
--raft-election-ms <n> 1000 Election timeout E: a follower that hears nothing from its leader for the leader lease (2E) plus a timeout drawn from [E, 2E] stands for election, 3–4 s at the default. Set the same on every meta node. Lower risks elections under load with no fault (at 500, a busy follower's delayed heartbeats caused them); higher makes a lost leader take longer to replace
--log-level <lvl> info
--tls-cert, --tls-key, --tls-ca unset mTLS

--listen vs --raft-advertise: --listen is what the pod binds to locally; --raft-advertise is what it tells peers to reach it at. In Kubernetes the chart sets advertise to the pod DNS (pod-0.meta-svc.ns.svc.cluster.local).

No --raft-bind flag exists — Raft RPC multiplexes over the same gRPC socket as MetadataService, so a Raft peer address is a node's gRPC port (9100), not its metrics port (9101).

The first Raft leader creates the admin user and every node writes its keys to <data-dir>/admin-creds.env.


iolake-osd

Object Storage Daemon.

Flags

iolake-osd \
  --listen 0.0.0.0:9200 \
  --meta-endpoint http://meta1:9100 \
  --disks /dev/sdb,/dev/sdc,/dev/sdd
Flag Default Purpose
-c, --config <path> /etc/objectio/osd.toml TOML config file, read if it exists; one that doesn't parse, or names a key the OSD doesn't know, stops the OSD
-l, --listen <addr> 0.0.0.0:9200 gRPC listen (the file's listen)
--advertise-addr <addr> none How peers reach this OSD (falls back to listen)
--disks <list> empty Explicit disk paths (comma-separated); disables discovery
--disk-filter <glob> empty Discovery glob(s), e.g. /dev/disk/by-id/wwn-*; repeatable
--disk-min-size <bytes> 1 GiB Lower bound for discovered disks
--init-blank-disks off Allow discovery to format blank disks (off by default)
--meta-endpoint <urls> http://localhost:9100 Meta nodes, comma-separated (the file's meta_endpoint)
--data-dir <path> none OSD metadata (WAL, object index); the file's [storage] data_dir, default ./osd-data
--metrics-port <n> 9201 Prometheus scrape port (0 none)
--scrub-interval-secs <n> 604800 Seconds between scrub passes; 0 off
--scrub-rate-mib <n> 50 Scrub read rate cap (MiB/s); 0 unthrottled
--full-ratio <r> 0.95 Share of each disk client writes may fill (env IOLAKE_OSD_FULL_RATIO)
--meta-cache-mib <n> 1024 Metadata index page cache (env IOLAKE_OSD_META_CACHE_MIB)
--meta-engine <e> native Metadata index engine: native, or rocksdb in a build with that feature (env IOLAKE_OSD_META_ENGINE)
--log-level <lvl> the file's [logging] level, else info RUST_LOG, when set, wins
--tls-cert, --tls-key, --tls-ca unset mTLS

TOML

# /etc/objectio/osd.toml
[osd]
listen = "0.0.0.0:9200"
advertise_addr = "http://osd1:9200"
meta_endpoint = "http://meta1:9100,http://meta2:9100,http://meta3:9100"

# Only here: no flag sets the failure domain. Unset levels are "default";
# host defaults to the machine's hostname.
[osd.failure_domain]
region = "eu-west-1"
datacenter = "dc1"
rack = "rack1"

[storage]
disks = ["/dev/sdb", "/dev/sdc", "/dev/sdd"]
# block_size: leave unset. A new disk's allocation unit, 4096 by default;
# every shard takes whole blocks, so a larger one wastes space on small
# objects.
data_dir = "/var/lib/objectio/osd"

[logging]
level = "info"

A config file that doesn't parse, or names a key the OSD doesn't know (a misspelt [osd.failure_domain], say), stops the OSD with the file's name and the error; with no file at the --config path, every setting is its default.

Disk discovery vs explicit disks. --disks is the production path — list the devices this OSD owns. --disk-filter is for fleets where device names aren't stable (use by-id paths). Without --init-blank-disks, unformatted disks are skipped rather than claimed, as a guard against accidental data loss. A disk named with --disks that can't be claimed (neither blank nor an IOLake disk) stops the OSD. The OSD records its disks by the ids in their superblocks: they may be named in any order and under other paths, but an OSD missing one of its disks doesn't start. A disk file that doesn't exist is created only on the OSD's first start; after that, it is a missing disk (runbook).


iolake-block-gateway

Block volumes over NBD and gRPC, stored as IOLake chunks.

iolake-block-gateway \
  --listen 0.0.0.0:9300 \
  --nbd-listen 0.0.0.0:10809 \
  --advertise-host block.internal \
  --meta-endpoint http://meta1:9100 \
  --data-dir /var/lib/objectio/block-gw
Flag Default Purpose
--listen <addr> 0.0.0.0:9300 BlockService gRPC listen. Under mTLS only a certificate of role admin (iolakectl's block commands) may call it
--nbd-listen <addr> 0.0.0.0:10809 NBD TCP listen. NBD has no authentication: anyone who reaches the port and knows a volume's name can attach it, so it must be reachable from the machines that attach volumes and nothing else. Listing the exports is refused
--advertise-host <host> empty Hostname used in NBD attachment URLs
--meta-endpoint <url> http://localhost:9100
--data-dir <path> ./block-gw-data The write journal (volumes, snapshots and chunk maps are kept in meta)
--metrics-listen <addr> 0.0.0.0:9301 Prometheus /metrics; empty serves none
--cache-bytes <n> 268435456 Write-cache size (256 MiB)
--flush-interval-s <n> 5 Background flush interval
--ec-k <n> / --ec-m <n> 4 / 2 EC profile for chunk objects: one with parity that every loss of m shards decodes (which), or the block gateway refuses to start
--log-level <lvl> info

iolake-aio

Single-process cluster (meta + OSD + gateway). See quickstart.md for the full usage.

Flag Default Purpose
--port <n> 9000 Public S3 / Iceberg / Delta / console port
--listen-addr <ip> 0.0.0.0 Bind interface
--data <dir> tempdir Persistent data root
--osds <n> 1 Number of OSDs to spawn in-process
--ec-k <n>, --ec-m <n> 0 Erasure-code the pool (needs --osds ≥ k+m); without them it is replication 1, however many OSDs
--disk-size <bytes> 0 (10 GiB sparse) Size of each OSD's disk file when created
--prometheus-url <url> empty Forwarded to the gateway
--external-endpoint <url> empty Forwarded to the gateway
--s3-domain <domain> none Forwarded to the gateway. Env IOLAKE_S3_DOMAIN.
--block-port <n> 0 (off) Run the block gateway too (NBD on --nbd-port, 10809)
--auth off Enable SigV4 on the gateway (admin AK/SK printed at startup)
--admin-port <n> 0 (off) Mirror of iolake-gateway --admin-listen — bind admin API + /metrics to its own port.
--ops-console-port <n> 0 (off) Mirror of --ops-console-listen — serve the ops SPA on its own port.
--tenant-console-port <n> 0 (off) Mirror of --tenant-console-listen — serve the tenant SPA on its own port.
--endpoint-name <name> empty Mirror of the gateway's --endpoint-name, for --port.
--data-port <port>=<name> — Mirror of --data-listen on --listen-addr. Repeatable.
--trusted-proxies <cidrs> empty Mirror of the gateway's --trusted-proxies.
--audit-log <path> unset Mirror of the gateway's --audit-log.
--memory-budget <size>, --max-requests <n>, --header-timeout-secs <n>, --body-timeout-secs <n> the gateway's Forwarded to the gateway
--log-level <lvl> info Honoured only if RUST_LOG is unset
--strict-port off Fail on --port conflict instead of picking ephemeral
--block-port <n> / --nbd-port <n> 0 (off) / 10809 Run the block gateway too, its gRPC and NBD on these ports
--block-listen-addr <ip> 127.0.0.1 Where the block gateway's two ports bind, whatever --listen-addr says
--tls-cert, --tls-key, --tls-ca unset mTLS for aio's own internal hops; its certificate names every role (mTLS)

Only the gateway's ports follow --listen-addr. Meta's gRPC and admin ports and the OSDs' bind loopback, and so does the block gateway unless --block-listen-addr says otherwise: NBD has no authentication, and without mTLS neither has the block gateway's gRPC.

Pass any of --admin-port / --ops-console-port / --tenant-console-port to test split-mode flows locally. The aio binary embeds both SPA bundles (ops + tenant) at compile time, so no separate console build is needed. With --strict-port set, all configured ports are checked for conflicts before launch.

Example — local split-mode test:

iolake-aio \
  --port 9000 \
  --admin-port 9001 \
  --ops-console-port 9002 \
  --tenant-console-port 9003 \
  --auth --strict-port \
  --data /tmp/iolake-aio

Admin AK/SK are written to <data>/meta/admin-creds.env on first start (export AWS_ACCESS_KEY_ID=… and export AWS_SECRET_ACCESS_KEY=… lines), and printed in the startup banner.


iolake-install

Sets a host up to run the services under systemd, and forms meta's Raft group. The Linux guide walks through it.

iolake-install init --role all --node-id 1 --advertise node1.internal \
  --meta-endpoint http://node1.internal:9100,http://node2.internal:9100,http://node3.internal:9100 \
  --disks /dev/disk/by-id/nvme-a,/dev/disk/by-id/nvme-b \
  --ec-profile 4+2 --systemd

iolake-install cluster init node1.internal node2.internal node3.internal

iolake-install disk list [--all]
iolake-install disk check /dev/sdb
iolake-install disk prepare /dev/sdb [--force] [--block-size <bytes>]

iolake-install status

init writes, for the roles it is given:

  • meta: the unit iolake-meta, its data directory /var/lib/objectio/meta;
  • OSD: for each disk i (in the order given), /etc/objectio/osd-<i>.toml, the unit iolake-osd-<i> and the data directory /var/lib/objectio/osd-<i>. One OSD per disk: an OSD holds at most one shard of a stripe;
  • gateway: the unit iolake-gateway, which also reads /etc/objectio/gateway.env when it exists (IOLAKE_MASTER_KEY=… for SSE, the same on every gateway).

The units start meta and the gateway with the flags the Helm chart passes, and each OSD with its file. Meta and the gateway run as the system user objectio, which init creates if it is missing; the OSDs run as root, to open raw devices. init formats no disk: each must be blank, which its OSD formats when it first starts, or an IOLake disk; anything else is refused. Run again, it rewrites the files it writes and keeps the data.

init flags:

Flag Default Purpose
--role <list> — all, meta, osd, gateway (comma-separated)
--disks <list> — The OSDs' disks, one OSD each (osd role)
--node-id <n> 1 This host's meta node id; each meta node its own (1, 2, 3 …)
--advertise <host> the hostname The name or address the other nodes reach this host at: meta's --raft-advertise, each OSD's advertise_addr
--meta-endpoint <urls> this host's meta Every meta node's gRPC address, comma-separated; required on a host without meta
--ec-profile <k+m> 4+2 The default pool's erasure code (--ec-k/--ec-m of meta and the gateway)
--replication <n> unset n copies instead of an erasure code (--replication of meta, --protection replication --replicas of the gateway)
--no-auth off The gateway without authentication (development only)
--region <name> us-east-1 The gateway's --region
--external-endpoint <url> unset The gateway's --external-endpoint
--s3-domain <domain> none The gateway's --s3-domain (repeatable, or comma-separated)
--meta-port, --meta-admin-port, --meta-metrics-port 9100, 9102, 9101 Meta's ports (metrics 0: none)
--osd-port, --osd-metrics-port 9200, 9201 OSD i's ports are these + 2i (metrics 0: none)
--gateway-port <n> 9000 The gateway's S3 port
--systemd off Install the units in /etc/systemd/system (and systemctl daemon-reload); without it, print the services' command lines
--bin-dir <dir> /usr/local/bin Where the IOLake binaries are
--root <dir> / Write everything under this directory instead, naming paths under it: a host's files, staged or tested elsewhere. No user is created and systemd is not reloaded

cluster init <node>… forms the Raft group from meta nodes' admin addresses (host, host:port, port 9102 by default, or a URL), the first being the one the group starts from: /init on it unless a group exists, then, on the leader, /add-learner for every node not yet a voter and /change-membership with all of them. Each node's id and Raft address come from its own /status. Run again, it changes nothing; a voter not named stops it, since dropping one is done by hand (meta HA). It waits up to --wait-secs (120) for every node to answer. Under mTLS give it --tls-cert, --tls-key (a certificate of role admin) and --tls-ca (the CA that signed the meta nodes'): it then speaks HTTPS to a node named without a scheme, and presents the certificate. The meta units init writes bind the admin port on every interface (--admin-host 0.0.0.0), so cluster init runs from any host.

disk prepare formats a disk ahead of its OSD, with the OSD's block size unless --block-size is given; it refuses a disk that isn't blank without --force, which destroys what the disk holds.


Configuration precedence

For every setting:

  1. CLI flag
  2. Environment variable (if the flag has one)
  3. The OSD's TOML file (OSD only)
  4. Built-in default

Logging is the exception: RUST_LOG, when set, wins over --log-level (and the file's [logging] level).

The Helm chart passes settings to meta and the gateway as flags in the container's command, and renders osd.toml for each OSD.