Configuration Reference¶
IOLake binaries are configured with CLI flags, some of which can also
be set by an environment variable (shown as [env: …] in each binary's
--help). Only iolake-osd reads a TOML file (--config); meta and
the gateway take flags only. A flag wins over its environment variable,
which wins over the OSD's file, which wins over the built-in default.
This page documents the flag surface for each binary against the
current main; <binary> --help is the complete list. Only the env
vars listed here are honoured.
Ports¶
Every binary's defaults are the cluster's ports, the ones the Helm chart and these guides use:
| Role | Port | Protocol |
|---|---|---|
| gateway S3 | 9000 | HTTP |
| meta gRPC (and Raft) | 9100 | gRPC |
| meta metrics | 9101 | HTTP |
| meta admin (Raft bootstrap / membership) | 9102 | HTTP; HTTPS under mTLS |
| OSD gRPC | 9200 | gRPC |
| OSD metrics | 9201 | HTTP |
| block-gateway gRPC | 9300 | gRPC |
| block-gateway NBD | 10809 | TCP (NBD) |
The clients' defaults follow: the gateway's, the OSD's and the block
gateway's --meta-endpoint default to http://localhost:9100. Beyond one
host, pass every address. A host running several OSDs gives each its own
ports: iolake-install puts OSD i on 9200 + 2i
and its metrics on 9201 + 2i.
---|---|---|---|
| gateway S3 | 9000 | 9000 | HTTP |
| meta gRPC | 9100 | 9001 | gRPC |
| meta metrics | 9101 | 9101 | HTTP |
| meta admin (Raft bootstrap / membership) | 9102 | 9102 | HTTP; HTTPS under mTLS |
| OSD gRPC | 9200 | none without a config file; 9002 when osd.toml has no listen | gRPC |
| OSD metrics | 9201 | 9201 | HTTP |
| block-gateway gRPC | 9300 | 9300 | gRPC |
| block-gateway NBD | 10809 | 10809 | TCP (NBD) |
| block-gateway metrics | 9301 | 9301 | HTTP |
Every port but the gateway's is internal: who may reach each is in the port table.
The chart's values are what you should document to end users. The code
defaults only matter if you run the binaries standalone with no
--listen override. The clients' defaults follow the code, not the
convention: the gateway's --meta-endpoint defaults to
http://localhost:9001, the block gateway's to http://localhost:9100.
Pass every address explicitly.
iolake-gateway¶
S3 / Iceberg / Delta Sharing / admin / console.
Flags¶
iolake-gateway \
--listen 0.0.0.0:9000 \
--meta-endpoint http://meta1:9100,http://meta2:9100,http://meta3:9100 \
--ec-k 4 --ec-m 2 \
--region us-east-1
| Flag | Default | Purpose |
|---|---|---|
-l, --listen <addr> |
0.0.0.0:9000 |
Data plane listen — S3 + Iceberg + Delta Sharing. In single-port mode (no split flags) also serves /_admin/*, /metrics and /_console/*. |
--admin-listen <addr> |
empty | Optional dedicated listener for /_admin/* + /metrics. Set this and admin moves off --listen entirely (see Listener split). |
--ops-console-listen <addr> |
empty | Optional dedicated listener for the ops console SPA + /_admin/* (system-admin surface). |
--tenant-console-listen <addr> |
empty | Optional dedicated listener for the tenant console SPA + /_admin/* (end-user surface). |
--meta-endpoint <urls> |
http://localhost:9100 |
Meta nodes: one address, or every meta node's, comma-separated. Any node serves any call (a follower forwards it to the Raft leader), and one that stops answering is skipped. See Meta high availability. |
--ec-k <n> |
4 |
EC data shards |
--ec-m <n> |
2 |
EC parity shards |
--protection <kind> |
ec |
ec, lrc, or replication |
--lrc-local-parity <n> |
0 |
LRC local parity count (with --protection=lrc) |
--lrc-global-parity <n> |
0 |
LRC global parity count |
--replicas <n> |
3 |
Replica count (with --protection=replication) |
--no-auth |
off | Disable SigV4 — dev only |
--prometheus-url <url> |
empty | A Prometheus that scrapes the cluster: the console's long ranges and iolakectl metrics query go through it. Without it /_admin/metrics/query answers that none is configured. Env IOLAKE_PROMETHEUS_URL. |
--no-reexport-metrics |
off | Leave the OSDs' and meta's metrics out of this gateway's /metrics; set it with more than one gateway (where to scrape). Env IOLAKE_NO_REEXPORT_METRICS. |
--no-bucket-metrics |
off | Leave per-bucket iolake_bucket_* series out of /metrics. Env IOLAKE_NO_BUCKET_METRICS. |
--inline-max-size <bytes> |
4096 |
Objects up to this size are stored in their metadata record; 0 turns it off |
--small-shard-max <bytes> |
16384 |
Shards up to this size travel with the object's metadata copies (at most 16384); 0 turns it off |
--endpoint-name <name> |
empty | Name of the --listen endpoint, seen by policies as aws:SourceVpce (see Public Access). Env IOLAKE_ENDPOINT_NAME. |
--data-listen <addr>=<name> |
— | A further data-plane listener (S3, Iceberg, Delta Sharing; no admin, no console) with its own endpoint name. Repeatable. |
--audit-log <path> |
unset | Append every request's audit event as JSON lines to this file (- for stdout). Env IOLAKE_AUDIT_LOG. |
--audit-spool <dir>, --audit-system-bucket <name> |
unset | The audit spool and the system's own audit bucket; the other --audit-* and --bucket-log-* flags are in audit and bucket logging |
--lifecycle-interval-secs <n> |
3600 |
How often the lifecycle worker scans buckets with rules |
--pack-interval-secs <n>, --pack-min-age-secs <n> |
0 (off), 3600 |
Small-object packing |
--replication-scan-secs <n> |
30 |
How often the replication scanner looks for unsent versions |
--trusted-proxies <cidrs> |
empty | Proxies whose X-Forwarded-For / X-Forwarded-Proto are believed for aws:SourceIp / aws:SecureTransport. Env IOLAKE_TRUSTED_PROXIES. Set it when TLS ends at an ingress. |
--memory-budget <size> |
half the memory limit, else 512MiB |
Memory for request bodies and GET read-ahead (512MiB, 2GiB); a request that doesn't fit gets 503 SlowDown. Unset: half the process's cgroup memory limit when it has one. Env IOLAKE_MEMORY_BUDGET. See gateway memory. |
--max-requests <n> |
1024 |
Requests served at once, each until its response is sent; more get 503 SlowDown. /health, /_ready and /metrics aren't counted. Env IOLAKE_MAX_REQUESTS. |
--header-timeout-secs <n> |
30 |
How long a request's headers may take to arrive, and an idle keep-alive connection waits for its next request. Env IOLAKE_HEADER_TIMEOUT_SECS. |
--body-timeout-secs <n> |
30 |
How long a request body may send no byte (400 RequestTimeout), and a client may read no byte of a response (disconnected). Env IOLAKE_BODY_TIMEOUT_SECS. |
--shutdown-delay-secs <n> |
5 |
On SIGTERM, how long /_ready answers 503 while the gateway still serves, before it stops taking connections. Env IOLAKE_SHUTDOWN_DELAY_SECS. See how a node stops. |
--shutdown-drain-secs <n> |
20 |
On shutdown, how long requests in flight are given to finish once the gateway stops taking connections. Env IOLAKE_SHUTDOWN_DRAIN_SECS. |
--region <name> |
us-east-1 |
The region requests are signed for (SigV4), and the one GetBucketLocation answers for every bucket |
--external-endpoint <url> |
empty | Public URL (Iceberg vended creds, Delta Sharing presign, OIDC callback) |
--s3-domain <domain> |
none | A domain clients address buckets under by host name, e.g. s3.example.com: photos.s3.example.com is bucket photos. Repeatable, or comma-separated. Env IOLAKE_S3_DOMAIN. Needs wildcard DNS and a wildcard certificate for the domain; see virtual-hosted addressing. |
--master-key-env <name> |
IOLAKE_MASTER_KEY |
Env var name holding the base64 SSE master key |
--kms-backend <kind> |
local |
local or vault |
--oidc-issuer-url <url> |
unset | Enables OIDC login for console + Iceberg JWTs |
--oidc-client-id <id> |
unset | OIDC client ID |
--oidc-client-secret <val> |
unset | OIDC client secret |
--oidc-audience <val> |
client id | JWT aud expected value |
--oidc-groups-claim <name> |
groups |
Claim mapping to group/role set |
--oidc-role-claim <name> |
role |
Claim mapping to single role |
--oidc-scopes <scopes> |
openid profile email |
Space-separated scope list |
--oidc-admin-roles <roles> |
empty | Comma-separated list of admin-granting roles |
--delta-access-key-id <ak> |
empty | Presign AK for Delta Sharing recipient URLs |
--delta-secret-key <sk> |
empty | Presign SK for Delta Sharing recipient URLs |
--delta-url-ttl-seconds <n> |
0 (3600) |
Lifetime of Delta Sharing presigned file URLs |
--host-provider <kind> |
noop |
noop (refuses host actions) or k8s — powers /_admin/hosts and /_admin/osds/*/reboot |
--host-provider-namespace <ns> |
default |
Namespace for the k8s provider (from $POD_NAMESPACE) |
--host-provider-osd-sts <name> |
iolake-osd |
OSD StatefulSet name for the k8s provider |
--topology-region <val> |
empty | Gateway's own topology — drives locality-aware reads |
--topology-zone <val> |
empty | |
--topology-datacenter <val> |
empty | |
--topology-rack <val> |
empty | |
--topology-host <val> |
empty | |
--log-level <lvl> |
info |
trace, debug, info, warn, error; RUST_LOG, when set, wins |
--tls-cert, --tls-key, --tls-ca |
unset | mTLS to meta and the OSDs (every binary has these). Env IOLAKE_TLS_CERT, IOLAKE_TLS_KEY, IOLAKE_TLS_CA. |
Environment variables¶
Besides those named in the table above, the clap parser reads these directly:
| Env | Flag |
|---|---|
IOLAKE_TOPOLOGY_REGION |
--topology-region |
IOLAKE_TOPOLOGY_ZONE |
--topology-zone |
IOLAKE_TOPOLOGY_DATACENTER |
--topology-datacenter |
IOLAKE_TOPOLOGY_RACK |
--topology-rack |
IOLAKE_TOPOLOGY_HOST |
--topology-host |
IOLAKE_HOST_PROVIDER |
--host-provider |
IOLAKE_HOST_PROVIDER_OSD_STS |
--host-provider-osd-sts |
POD_NAMESPACE |
--host-provider-namespace |
Read separately from the flag surface:
| Env | Purpose |
|---|---|
IOLAKE_MASTER_KEY |
32-byte base64 SSE-S3 master key |
IOLAKE_CONSOLE_DIR |
Directory holding both console bundles, ops/ and tenant/ (default /usr/share/objectio/console): the base for the two below. |
IOLAKE_OPS_CONSOLE_DIR |
Path to the ops SPA bundle. Default: $IOLAKE_CONSOLE_DIR/ops. Served on --ops-console-listen. |
IOLAKE_TENANT_CONSOLE_DIR |
Path to the tenant SPA bundle. Default: $IOLAKE_CONSOLE_DIR/tenant. Served on --tenant-console-listen. |
RUST_LOG |
Override --log-level; tracing directive syntax |
Listener split¶
By default the gateway is a single-port deployment: --listen carries
S3, Iceberg, Delta Sharing, /_admin/*, /metrics, and /_console/*
all on the same socket. This matches MinIO's default and is fine for
small clusters.
For larger or regulated deployments you can split the four planes
across separate listeners by setting any combination of
--admin-listen, --ops-console-listen, --tenant-console-listen.
Each binds independently; you can enable one, two, or all three.
| Listener | Example | Routes mounted | Audience |
|---|---|---|---|
| data | --listen 0.0.0.0:9000 |
/, /{bucket}*, /iceberg/v1/*, /api/2.1/unity-catalog/*, /delta-sharing/v1/*, /health |
Public — SDK / CLI / Spark / Trino |
| admin API | --admin-listen 127.0.0.1:9001 |
/_admin/*, /metrics, /health |
CLI, third-party integrations (Datadog, billing, OTel) |
| ops console | --ops-console-listen 127.0.0.1:9002 |
/_console/* (ops SPA), /_console/api/*, /_admin/*, /health |
System operators on a mgmt VLAN |
| tenant console | --tenant-console-listen 0.0.0.0:9003 |
/_console/* (tenant SPA), /_console/api/*, /_admin/*, /health |
End users — safe to expose publicly |
Behavior in split mode:
/_admin/*is mounted on the admin API listener AND on each console listener (the browser stays same-origin — no CORS). It is removed from the data listener —/_admin/*requests there fall through the S3 wildcard and return SigV4 errors./metricslives on the admin API listener only./_console/*is removed from the data listener; each console listener serves its own SPA bundle fromIOLAKE_OPS_CONSOLE_DIR/IOLAKE_TENANT_CONSOLE_DIR.- The startup log emits a
WARNif--admin-listenor--ops-console-listenis bound to0.0.0.0— the intent of the split is to firewall the control plane to a mgmt interface. - The same
/_admin/*handlers run on every listener that mounts them. Per-tenant scope is enforced server-side (seerequire_tenant_admin_access), so the tenant console listener is not a privilege-elevation path even though it exposes the same surface as the ops console.
Why two consoles, not one with role-gating: see architecture/console-split.
Example — public S3 on the WAN, control plane on a mgmt VLAN:
iolake-gateway \
--listen 0.0.0.0:9000 \
--admin-listen 10.0.5.10:9001 \
--ops-console-listen 10.0.5.10:9002 \
--tenant-console-listen 0.0.0.0:9003 \
--meta-endpoint http://meta1:9100 \
--region us-east-1
Omit all three split flags and the gateway serves everything on
--listen.
iolake-meta¶
Raft metadata cluster.
Flags¶
iolake-meta \
--node-id 1 \
--listen 0.0.0.0:9100 \
--raft-advertise meta1:9100 \
--data-dir /var/lib/objectio/meta
A meta node starts outside any cluster. The Raft group is formed over
the admin port: POST /init on the first node, then on the leader
POST /add-learner for each other node ({"node_id": 2, "addr":
"meta2:9100"}, the address being the peer's gRPC port, 9100) and
POST /change-membership with every voter ({"voters": [1, 2, 3]}).
A single node needs only /init. See Linux
and meta HA.
| Flag | Default | Purpose |
|---|---|---|
--node-id <n> |
1 |
Raft node ID; each node of a group needs its own |
-l, --listen <addr> |
0.0.0.0:9100 |
gRPC listen — MetadataService + Raft RPC |
--ec-k <n> |
4 |
Default EC data shards |
--ec-m <n> |
2 |
Default EC parity shards, at least 1. With --ec-k, a profile every loss of m shards decodes (which); meta refuses to start with another, 1+0 too |
--replication <n> |
unset | Use replication instead of EC: n copies of each object, from 1 (one copy, no redundancy: a one-OSD dev cluster) |
--data-dir <path> |
/var/lib/objectio/meta |
Persistent Raft + redb state |
--admin-user <name> |
admin |
Initial admin username (created on first startup) |
--metrics-port <n> |
9101 |
Prometheus scrape port (0 none) |
--admin-port <n> |
9102 |
Admin port — Raft bootstrap + membership (/init, /add-learner, /change-membership) and the node's state (/status, and /health and /ready for probes). HTTPS under mTLS, where every call but reading the state takes a client certificate of role admin, and reading it takes any client, with a certificate or none; plain and unauthenticated without |
--admin-host <ip> |
the host of --listen |
Address the admin port binds: with --listen 0.0.0.0:… every interface, with loopback this host only |
--raft-advertise <addr> |
empty | Address advertised to Raft peers (defaults to --listen) |
--repair-interval-secs <n> |
3600 |
Seconds between repair passes (leader); 0 off |
--drain-interval-secs <n>, --drain-batch <n> |
30, 64 |
Drain sweeps, and shards moved per draining OSD per sweep |
--min-free-mib <n> |
1024 |
Free space meta keeps on the filesystem of --data-dir: below it the leader refuses writes (507) and a follower takes its leader's entries down to half of it; reads go on, and writes resume once there is room (meta's disk filling) |
--raft-snapshot-every <n> |
500000 |
Log entries between snapshots; the log is compacted into each |
--raft-keep-logs <n> |
10000 |
Entries kept behind the last snapshot |
--raft-checkpoint-ms <n> |
200 |
How often applied state is made durable |
--raft-heartbeat-ms <n> |
500 |
Raft heartbeat interval; also how long the leader waits for a follower to take entries |
--raft-election-ms <n> |
1000 |
Election timeout E: a follower that hears nothing from its leader for the leader lease (2E) plus a timeout drawn from [E, 2E] stands for election, 3–4 s at the default. Set the same on every meta node. Lower risks elections under load with no fault (at 500, a busy follower's delayed heartbeats caused them); higher makes a lost leader take longer to replace |
--log-level <lvl> |
info |
|
--tls-cert, --tls-key, --tls-ca |
unset | mTLS |
--listen vs --raft-advertise: --listen is what the pod binds to
locally; --raft-advertise is what it tells peers to reach it at. In
Kubernetes the chart sets advertise to the pod DNS
(pod-0.meta-svc.ns.svc.cluster.local).
No --raft-bind flag exists — Raft RPC multiplexes over the same
gRPC socket as MetadataService, so a Raft peer address is a node's
gRPC port (9100), not its metrics port (9101).
The first Raft leader creates the admin user and every node writes its
keys to <data-dir>/admin-creds.env.
iolake-osd¶
Object Storage Daemon.
Flags¶
iolake-osd \
--listen 0.0.0.0:9200 \
--meta-endpoint http://meta1:9100 \
--disks /dev/sdb,/dev/sdc,/dev/sdd
| Flag | Default | Purpose |
|---|---|---|
-c, --config <path> |
/etc/objectio/osd.toml |
TOML config file, read if it exists; one that doesn't parse, or names a key the OSD doesn't know, stops the OSD |
-l, --listen <addr> |
0.0.0.0:9200 |
gRPC listen (the file's listen) |
--advertise-addr <addr> |
none | How peers reach this OSD (falls back to listen) |
--disks <list> |
empty | Explicit disk paths (comma-separated); disables discovery |
--disk-filter <glob> |
empty | Discovery glob(s), e.g. /dev/disk/by-id/wwn-*; repeatable |
--disk-min-size <bytes> |
1 GiB | Lower bound for discovered disks |
--init-blank-disks |
off | Allow discovery to format blank disks (off by default) |
--meta-endpoint <urls> |
http://localhost:9100 |
Meta nodes, comma-separated (the file's meta_endpoint) |
--data-dir <path> |
none | OSD metadata (WAL, object index); the file's [storage] data_dir, default ./osd-data |
--metrics-port <n> |
9201 |
Prometheus scrape port (0 none) |
--scrub-interval-secs <n> |
604800 |
Seconds between scrub passes; 0 off |
--scrub-rate-mib <n> |
50 |
Scrub read rate cap (MiB/s); 0 unthrottled |
--full-ratio <r> |
0.95 |
Share of each disk client writes may fill (env IOLAKE_OSD_FULL_RATIO) |
--meta-cache-mib <n> |
1024 |
Metadata index page cache (env IOLAKE_OSD_META_CACHE_MIB) |
--meta-engine <e> |
native |
Metadata index engine: native, or rocksdb in a build with that feature (env IOLAKE_OSD_META_ENGINE) |
--log-level <lvl> |
the file's [logging] level, else info |
RUST_LOG, when set, wins |
--tls-cert, --tls-key, --tls-ca |
unset | mTLS |
TOML¶
# /etc/objectio/osd.toml
[osd]
listen = "0.0.0.0:9200"
advertise_addr = "http://osd1:9200"
meta_endpoint = "http://meta1:9100,http://meta2:9100,http://meta3:9100"
# Only here: no flag sets the failure domain. Unset levels are "default";
# host defaults to the machine's hostname.
[osd.failure_domain]
region = "eu-west-1"
datacenter = "dc1"
rack = "rack1"
[storage]
disks = ["/dev/sdb", "/dev/sdc", "/dev/sdd"]
# block_size: leave unset. A new disk's allocation unit, 4096 by default;
# every shard takes whole blocks, so a larger one wastes space on small
# objects.
data_dir = "/var/lib/objectio/osd"
[logging]
level = "info"
A config file that doesn't parse, or names a key the OSD doesn't know
(a misspelt [osd.failure_domain], say), stops the OSD with the file's
name and the error; with no file at the --config path, every setting
is its default.
Disk discovery vs explicit disks. --disks is the production
path — list the devices this OSD owns. --disk-filter is for fleets
where device names aren't stable (use by-id paths). Without
--init-blank-disks, unformatted disks are skipped rather than
claimed, as a guard against accidental data loss. A disk named with
--disks that can't be claimed (neither blank nor an IOLake disk)
stops the OSD. The OSD records its disks by the ids in their
superblocks: they may be named in any order and under other paths, but
an OSD missing one of its disks doesn't start. A disk file that doesn't
exist is created only on the OSD's first start; after that, it is a
missing disk
(runbook).
iolake-block-gateway¶
Block volumes over NBD and gRPC, stored as IOLake chunks.
iolake-block-gateway \
--listen 0.0.0.0:9300 \
--nbd-listen 0.0.0.0:10809 \
--advertise-host block.internal \
--meta-endpoint http://meta1:9100 \
--data-dir /var/lib/objectio/block-gw
| Flag | Default | Purpose |
|---|---|---|
--listen <addr> |
0.0.0.0:9300 |
BlockService gRPC listen. Under mTLS only a certificate of role admin (iolakectl's block commands) may call it |
--nbd-listen <addr> |
0.0.0.0:10809 |
NBD TCP listen. NBD has no authentication: anyone who reaches the port and knows a volume's name can attach it, so it must be reachable from the machines that attach volumes and nothing else. Listing the exports is refused |
--advertise-host <host> |
empty | Hostname used in NBD attachment URLs |
--meta-endpoint <url> |
http://localhost:9100 |
|
--data-dir <path> |
./block-gw-data |
The write journal (volumes, snapshots and chunk maps are kept in meta) |
--metrics-listen <addr> |
0.0.0.0:9301 |
Prometheus /metrics; empty serves none |
--cache-bytes <n> |
268435456 |
Write-cache size (256 MiB) |
--flush-interval-s <n> |
5 |
Background flush interval |
--ec-k <n> / --ec-m <n> |
4 / 2 |
EC profile for chunk objects: one with parity that every loss of m shards decodes (which), or the block gateway refuses to start |
--log-level <lvl> |
info |
iolake-aio¶
Single-process cluster (meta + OSD + gateway). See quickstart.md for the full usage.
| Flag | Default | Purpose |
|---|---|---|
--port <n> |
9000 |
Public S3 / Iceberg / Delta / console port |
--listen-addr <ip> |
0.0.0.0 |
Bind interface |
--data <dir> |
tempdir | Persistent data root |
--osds <n> |
1 |
Number of OSDs to spawn in-process |
--ec-k <n>, --ec-m <n> |
0 |
Erasure-code the pool (needs --osds ≥ k+m); without them it is replication 1, however many OSDs |
--disk-size <bytes> |
0 (10 GiB sparse) |
Size of each OSD's disk file when created |
--prometheus-url <url> |
empty | Forwarded to the gateway |
--external-endpoint <url> |
empty | Forwarded to the gateway |
--s3-domain <domain> |
none | Forwarded to the gateway. Env IOLAKE_S3_DOMAIN. |
--block-port <n> |
0 (off) |
Run the block gateway too (NBD on --nbd-port, 10809) |
--auth |
off | Enable SigV4 on the gateway (admin AK/SK printed at startup) |
--admin-port <n> |
0 (off) |
Mirror of iolake-gateway --admin-listen — bind admin API + /metrics to its own port. |
--ops-console-port <n> |
0 (off) |
Mirror of --ops-console-listen — serve the ops SPA on its own port. |
--tenant-console-port <n> |
0 (off) |
Mirror of --tenant-console-listen — serve the tenant SPA on its own port. |
--endpoint-name <name> |
empty | Mirror of the gateway's --endpoint-name, for --port. |
--data-port <port>=<name> |
— | Mirror of --data-listen on --listen-addr. Repeatable. |
--trusted-proxies <cidrs> |
empty | Mirror of the gateway's --trusted-proxies. |
--audit-log <path> |
unset | Mirror of the gateway's --audit-log. |
--memory-budget <size>, --max-requests <n>, --header-timeout-secs <n>, --body-timeout-secs <n> |
the gateway's | Forwarded to the gateway |
--log-level <lvl> |
info |
Honoured only if RUST_LOG is unset |
--strict-port |
off | Fail on --port conflict instead of picking ephemeral |
--block-port <n> / --nbd-port <n> |
0 (off) / 10809 |
Run the block gateway too, its gRPC and NBD on these ports |
--block-listen-addr <ip> |
127.0.0.1 |
Where the block gateway's two ports bind, whatever --listen-addr says |
--tls-cert, --tls-key, --tls-ca |
unset | mTLS for aio's own internal hops; its certificate names every role (mTLS) |
Only the gateway's ports follow --listen-addr. Meta's gRPC and admin
ports and the OSDs' bind loopback, and so does the block gateway unless
--block-listen-addr says otherwise: NBD has no authentication, and
without mTLS neither has the block gateway's gRPC.
Pass any of --admin-port / --ops-console-port / --tenant-console-port
to test split-mode flows locally. The aio binary embeds both SPA
bundles (ops + tenant) at compile time, so no separate console build
is needed. With --strict-port set, all configured ports are
checked for conflicts before launch.
Example — local split-mode test:
iolake-aio \
--port 9000 \
--admin-port 9001 \
--ops-console-port 9002 \
--tenant-console-port 9003 \
--auth --strict-port \
--data /tmp/iolake-aio
Admin AK/SK are written to <data>/meta/admin-creds.env on first start
(export AWS_ACCESS_KEY_ID=… and export AWS_SECRET_ACCESS_KEY=…
lines), and printed in the startup banner.
iolake-install¶
Sets a host up to run the services under systemd, and forms meta's Raft group. The Linux guide walks through it.
iolake-install init --role all --node-id 1 --advertise node1.internal \
--meta-endpoint http://node1.internal:9100,http://node2.internal:9100,http://node3.internal:9100 \
--disks /dev/disk/by-id/nvme-a,/dev/disk/by-id/nvme-b \
--ec-profile 4+2 --systemd
iolake-install cluster init node1.internal node2.internal node3.internal
iolake-install disk list [--all]
iolake-install disk check /dev/sdb
iolake-install disk prepare /dev/sdb [--force] [--block-size <bytes>]
iolake-install status
init writes, for the roles it is given:
- meta: the unit
iolake-meta, its data directory/var/lib/objectio/meta; - OSD: for each disk i (in the order given),
/etc/objectio/osd-<i>.toml, the unitiolake-osd-<i>and the data directory/var/lib/objectio/osd-<i>. One OSD per disk: an OSD holds at most one shard of a stripe; - gateway: the unit
iolake-gateway, which also reads/etc/objectio/gateway.envwhen it exists (IOLAKE_MASTER_KEY=…for SSE, the same on every gateway).
The units start meta and the gateway with the flags the Helm chart passes,
and each OSD with its file. Meta and the gateway run as the system user
objectio, which init creates if it is missing; the OSDs run as root, to
open raw devices. init formats no disk: each must be blank, which its OSD
formats when it first starts, or an IOLake disk; anything else is
refused. Run again, it rewrites the files it writes and keeps the data.
init flags:
| Flag | Default | Purpose |
|---|---|---|
--role <list> |
— | all, meta, osd, gateway (comma-separated) |
--disks <list> |
— | The OSDs' disks, one OSD each (osd role) |
--node-id <n> |
1 |
This host's meta node id; each meta node its own (1, 2, 3 …) |
--advertise <host> |
the hostname | The name or address the other nodes reach this host at: meta's --raft-advertise, each OSD's advertise_addr |
--meta-endpoint <urls> |
this host's meta | Every meta node's gRPC address, comma-separated; required on a host without meta |
--ec-profile <k+m> |
4+2 |
The default pool's erasure code (--ec-k/--ec-m of meta and the gateway) |
--replication <n> |
unset | n copies instead of an erasure code (--replication of meta, --protection replication --replicas of the gateway) |
--no-auth |
off | The gateway without authentication (development only) |
--region <name> |
us-east-1 |
The gateway's --region |
--external-endpoint <url> |
unset | The gateway's --external-endpoint |
--s3-domain <domain> |
none | The gateway's --s3-domain (repeatable, or comma-separated) |
--meta-port, --meta-admin-port, --meta-metrics-port |
9100, 9102, 9101 |
Meta's ports (metrics 0: none) |
--osd-port, --osd-metrics-port |
9200, 9201 |
OSD i's ports are these + 2i (metrics 0: none) |
--gateway-port <n> |
9000 |
The gateway's S3 port |
--systemd |
off | Install the units in /etc/systemd/system (and systemctl daemon-reload); without it, print the services' command lines |
--bin-dir <dir> |
/usr/local/bin |
Where the IOLake binaries are |
--root <dir> |
/ |
Write everything under this directory instead, naming paths under it: a host's files, staged or tested elsewhere. No user is created and systemd is not reloaded |
cluster init <node>… forms the Raft group from meta nodes' admin
addresses (host, host:port, port 9102 by default, or a URL), the
first being the one the group starts from: /init on it unless a group
exists, then, on the leader, /add-learner for every node not yet a
voter and /change-membership with all of them. Each node's id and Raft
address come from its own /status. Run again, it changes nothing; a
voter not named stops it, since dropping one is done by hand
(meta HA). It waits up to --wait-secs
(120) for every node to answer. Under mTLS
give it --tls-cert, --tls-key (a certificate of role admin) and
--tls-ca (the CA that signed the meta nodes'): it then speaks HTTPS to
a node named without a scheme, and presents the certificate. The meta
units init writes bind the admin port on every interface
(--admin-host 0.0.0.0), so cluster init runs from any host.
disk prepare formats a disk ahead of its OSD, with the OSD's block size
unless --block-size is given; it refuses a disk that isn't blank without
--force, which destroys what the disk holds.
Configuration precedence¶
For every setting:
- CLI flag
- Environment variable (if the flag has one)
- The OSD's TOML file (OSD only)
- Built-in default
Logging is the exception: RUST_LOG, when set, wins over --log-level
(and the file's [logging] level).
The Helm chart passes settings to meta and the gateway as flags in the
container's command, and renders osd.toml for each OSD.