Running IOLake in production¶

What a production cluster needs, how to size it, what to alert on, and what to do when each alert fires. Read it with the failure-recovery, meta HA, metrics and upgrades pages, which go deeper on each subject.

Before going live¶

Every item is a default you have to change, or a requirement the software can't check for you.

	Requirement	Why
Redundancy	An erasure code: the chart's default is 4+2 (`replication: 0`, `ec.k`/`ec.m`), and it refuses to render with fewer OSDs than k + m; on hosts, `--ec-k`/`--ec-m`. 4+2 for 7–12 hosts, 8+4 from 13; these profiles only.	Without parity (or with one copy), one lost disk loses data.
Hosts	At least k + m OSD hosts, one more to rebuild onto (7 for 4+2).	With fewer, placement puts several shards of a stripe on one host, and losing that host can lose data. Placement doesn't refuse it.
Meta	3 meta nodes (5 for two failures at once), each on its own host.	Raft needs a majority: 3 survive one loss.
Authentication	On: `gateway.noAuth: false`, the chart's default.	Without it anyone who reaches the gateway can read and write everything.
mTLS	`tls.enabled: true` (mTLS).	Shards and metadata cross the network between services.
Drives	NVMe with power-loss protection, or a UPS for the whole host.	An acknowledged write is fsynced; on a drive with a volatile write cache and no PLP, a power cut can still lose it.
OSD layout	Shards on the raw device; each OSD's state directory (`--data-dir`: metadata WAL and index) on a small partition of the same drive (about 2%), not the boot disk.	Every metadata write fsyncs its WAL. With 18 OSDs on a host, a shared boot disk becomes the bottleneck, and its loss takes every OSD.
Clocks	NTP (chrony) on every host, offset well under 100 ms.	A gateway whose clock is more than 500 ms from meta's refuses writes (503) until it is back in step: write order depends on clocks.
Monitoring	Prometheus scraping the gateways, meta and OSDs, the alert rules loaded, and the alerts routed to someone.	Repair is automatic; deciding that a host is gone for good is not.
Audit	`gateway.audit` with a system bucket, if you need an audit trail (audit).	Events are spooled to disk and delivered later, so a receiver outage loses nothing.

Sizing¶

Hardware design and sizing goes from a capacity and throughput requirement to a bill of quantities, with a calculator: hosts, drives, CPU, memory, network, meta and gateways. In short:

Capacity: usable ≈ raw × k/(k+m) × 0.95, less one host's worth for rebuilds.
Memory: about 1.5 GB per OSD (one per drive), flat however many objects it holds; 64 GB for a host of 18 drives.
Metadata: about 1.5 KB per object copy in each OSD's index, on a partition of the same drive; about 750 bytes per object on each meta node.
Meta's volume: 3.2 × its database plus 1 GiB: the database, its Raft log, the snapshot it keeps and, while the next one is built, that one too, and the reserve meta keeps free (--min-free-mib, 1 GiB). Under the reserve, meta takes no writes (meta's disk filling). Give it a volume of its own.

Gateway memory and overload¶

A gateway's memory is bounded by what it lets in, not by the load it is offered. Under too much load it answers 503 SlowDown (with Retry-After: 1), which S3 SDKs retry with backoff, rather than queueing without end or running out of memory.

GET streams. An object is read, decoded and sent one stripe at a time, the next stripe read while the client takes the last, so a GET of any size holds about two stripes. The first stripe is read before the response headers are sent: a failure there is still an error status. A stripe that can't be read after the headers ends the connection short of Content-Length, never with wrong bytes, and is counted in iolake_gateway_get_aborted_total. A client sees a short read, which SDKs retry.
Large uploads stream. A PUT or an UploadPart over 16 MiB (up to S3's 5 GiB) is never held whole: its body is erasure-coded and written stripe by stripe as it arrives, two stripes in flight while the next is read, so an upload of any size holds about three stripes (48 MiB with 4+2). It is committed only once all of it has arrived and passed its checks (large uploads).
A memory budget (--memory-budget) holds request bodies and GET read-ahead. A body read whole (an upload up to 16 MiB; any other request, at most 100 MiB) reserves its whole declared length before a byte of it is read, after the request is authenticated; a streamed upload reserves the stripes it holds at once (or its length, if less); a GET reserves each stripe before reading it and frees it once the client has been sent it. A new request that doesn't fit gets 503 SlowDown; a GET already sending waits for room for its next stripe, up to the body timeout. The default is half the process's memory limit (its cgroup's: a container's resources.limits.memory), or 512 MiB without one. The budget counts what requests bring in and take out, not every buffer that handles them (erasure coding's shards, socket buffers), which is why it is half the limit and not all of it.
An in-flight limit (--max-requests, 1024): a request holds a slot until its response has been sent; beyond it, 503 SlowDown.
Timeouts. Request headers must arrive within --header-timeout-secs (30), which is also how long an idle keep-alive connection is kept. A request body that sends no byte for --body-timeout-secs (30) gets 400 RequestTimeout; a client that reads no byte of a response for as long is disconnected, and what its GET held is freed.

/health, /_ready and /metrics are outside every limit, so a probe or a scrape always reaches an overloaded gateway.

To size it: a gateway needs about twice its budget, plus its baseline (about 100 MB). Give each gateway a memory limit; the budget follows it. Steady SlowDowns, seen in iolake_gateway_requests_rejected_total by reason, mean the gateway is short of memory (memory) or of request slots (requests): add gateways, or raise the limit. The budget in use is iolake_gateway_memory_budget_in_use_bytes.

Alerts¶

The chart ships the rules in deploy/helm/iolake/alerts/iolake-rules.yaml: - its own Prometheus loads them when monitoring.enabled is set; - monitoring.prometheusRule.enabled also publishes them as a PrometheusRule for a prometheus-operator.

Each alert has a runbook below.

Alert	Severity	Runbook
IOLakeObjectsUnreadable	critical	Objects unreadable
IOLakeUnrecoverableStripes	critical	Objects unreadable
IOLakeObjectsAtRisk	warning	OSD down
IOLakeRepairNotRunning	warning	Repair not running
IOLakeScrubStale	warning	Repair not running
IOLakeCorruptShards	warning	Corrupt shards
IOLakePgDown	critical	Placement groups
IOLakePgIncomplete	critical	Placement groups
IOLakeObjectsUnfound	critical	Objects unreadable
IOLakeObjectsLost	critical	Objects unreadable
IOLakePgNotCleanWithOsdsUp	warning	Placement groups
IOLakePgUndersized	warning	Placement groups
IOLakePgScrubStale	warning	Repair not running
IOLakeMetaNoLeader	critical	Meta
IOLakeMetaElectionsChurn	warning	Meta
IOLakeMetaFollowerLagging	warning	Meta
IOLakeHealQueueFailing	warning	Meta
IOLakeMetaDiskLow	warning	Disks filling
IOLakeMetaDiskFull	critical	Disks filling
IOLakeGatewayClockSkewed	critical	Clock skew
IOLakeOsdDown	warning	OSD down
IOLakeDiskNearFull	warning	Disks filling
IOLakeDiskFull	critical	Disks filling
IOLakeClusterFilling	warning	Disks filling
IOLakeSlowShardSync	warning	Slow disk
IOLakeS3ServerErrors	warning	5xx
IOLakeSpaceLeaking	info	5xx
IOLakeAuditEventsDropped	warning	Audit
IOLakeReplicationLagging	warning	Replication
IOLakeSlowBlockJournal, IOLakeBlockFlushFailing, IOLakeBlockReadErrors		Block

Not covered by any alert yet: certificate expiry (no metric; watch cert-manager's own) and the audit spool's size.

Runbooks¶

Commands are iolakectl (a SigV4 client of the gateway's admin API, bin/cli/README.md in the code repo) and meta's admin port (9102; under mTLS HTTPS, and membership changes take a certificate of role admin).

An OSD is down¶

What happens on its own. - Meta probes every OSD every 15 s. Three missed probes, about 45 s, mark it Down. - For the first 120 s it is down (config pg/down_out_seconds), its placement groups keep it: writes go on degraded while each stripe still reaches its write quorum on distinct OSDs (k + 1 shards, two copies, or all but g − 1 shards of an LRC stripe; 503 short of it), and reads decode around it. A restart or a reboot moves nothing. - After that, each of its placement groups gets a stand-in OSD (within the pool's placement rule) and backfills onto it: the PG is rebuilt to full redundancy without the down OSD. A position no OSD can take leaves the PG Undersized.

What you do. 1. iolakectl node list: which OSD, and since when. 2. If the process died: restart it (systemctl restart iolake-osd, or the pod). Back within 120 s, it rejoins and recovery brings it up to date with what was written while it was gone. 3. If the host or drive is gone for good: iolakectl osd set-state <node_id> out, so it is no longer counted as in service. Its PGs already have stand-ins; watch them backfill (iolakectl pg list --state Degraded). 4. If it won't start, and its log says its metadata log is damaged ("damage, not a torn append", or records missing): what it holds can't be trusted. Treat it as a failed drive (step 3, then replace the disk with a fresh state directory). A crash alone leaves a torn tail, which the OSD cuts off by itself.

An OSD whose metadata log failed a write or an fsync ("refusing every write until it is reopened" in its log) stays up but refuses every write. Restart it: the restart finds where the log really ends. If it happens again, the drive is failing; replace it.

Done when IOLakeObjectsAtRisk clears and every placement group is Clean again (iolake_meta_pgs_not_clean 0).

Replacing a disk¶

If the disk still reads: iolakectl osd set-state <node_id> draining, and wait for iolakectl cluster drain-status to show it empty. It turns out by itself.
If it doesn't read: iolakectl osd set-state <node_id> out.
Swap the drive and start the OSD on it with the same state directory: it keeps its identity, finds the disk blank, and forgets the shards it held there. On an OSD with several disks, the blank disk takes the place of the one that is missing, whatever its path. A file-backed disk (disk.raw on a volume) is replaced by a blank file the size of the old one, made before the OSD starts (truncate -s <size> /data/disk0/disk.raw): an OSD makes a disk file itself only on its first start.
iolakectl osd set-state <node_id> in. Repair rebuilds what it held.

An OSD that won't start for a missing disk¶

An OSD knows its disks by the ids in their superblocks, not by their paths or order. Started without one of them, it doesn't start, and its log says which (this OSD's disk 1 (id …, last at /dev/…) is missing). Running on the rest would look for shards on the wrong disks.

The disk is there under another name (a renamed device, a glob that missed it): name it again, at any path, in any order.
A disk file does not exist (disk … does not exist, and this OSD has run on its disks): its volume isn't mounted. Mount it; if the disk is lost, put a blank file there as above.
The disk is dead: replace it with a blank one, as above.
No replacement, and the OSD should run on the disks it has left: that is a new OSD. Set the old one out, wait until iolakectl cluster drain-status shows it evacuated, then wipe its state directory and its remaining disks and start it on them: it joins as a new OSD.

It also refuses a disk that is not blank and not one of its own (another OSD's, or one replaced earlier and put back): remove it from the OSD's disks, or wipe it to add it as a new disk. A disk named with --disks that can't be read as an IOLake disk stops the OSD, rather than being skipped.

Disks filling or full¶

Near full (85%). Plan capacity: add OSDs or hosts (the balancer moves data onto them), or delete data. Check iolakectl cluster usage.
Full (95%). That OSD refuses client writes and clients get 507. The space above 95% is kept for repair. Other OSDs keep taking writes for stripes not placed on the full one.
Find which disks: iolake_disk_used_bytes / iolake_disk_capacity_bytes.
One full disk among emptier ones is uneven placement: the balancer evens it out (iolakectl cluster rebalance status).
Everything full: add capacity. Deletes free space within a minute.
Don't raise --full-ratio to get writes going: it spends the room repair needs after the next failure.
The filesystem under an OSD full (its sparse disk files, or its state directory): writes it holds get 507 and the OSD logs no room. Free space on that filesystem, or grow it; writes resume without a restart. Keep disk files on a filesystem with room for their full size, or on raw devices. If the OSD logs that its metadata log takes no writes after an earlier failure, an fsync failed: restart it once there is room.
Meta's disk low or full (IOLakeMetaDiskLow: under twice the reserve for 10 minutes; IOLakeMetaDiskFull: under the reserve). Under the reserve (--min-free-mib, 1 GiB) the meta leader refuses writes that need meta, 507, and a follower falls behind; reads and object PUTs go on (meta's disk filling).
Which node, and why: curl http://<meta>:9102/status on each, for disk_free, disk_reserve and disk_full.
Grow the meta volume (on Kubernetes, the meta PVC, if its storage class allows expansion), or free space on that filesystem if something else shares it. Don't delete files under meta's data directory. Writes resume by themselves.
A meta node that exited with raft checkpoint failed or Raft stopped and a No space left on device in its log met a full filesystem anyway (something else took the reserve): it starts again, and catches up, once there is room.
Don't lower --min-free-mib to get writes going: it spends the room meta needs to finish what it has under way.

A gateway refuses writes for clock skew¶

Its clock is more than 500 ms from meta's (iolake_gateway_clock_offset_seconds gives the offset). Writes through it fail with 503 (clients retry, ideally through another gateway). Reads work.

Fix time sync on that host (chronyc tracking, chronyc makestep). Writes resume within a second of the clock coming back in step. Don't restart the gateway: it doesn't help.

Meta has no leader, or elects often¶

No leader for more than a minute: fewer than a majority of meta nodes are up or can reach each other. Writes fail with 503; reads of object data mostly still work.
curl http://<meta>:9102/status on each node shows its role, its term, and who it thinks leads.
Bring a majority back. A node lost for good is replaced with POST :9102/add-learner then POST :9102/change-membership (meta HA).
No procedure exists yet for a majority lost for good.
Elections churn: usually a node that drops off the network intermittently, or a disk too slow for its heartbeats. Compare the nodes' iolake_meta_commit_seconds.
A follower lags: it is catching up, or its disk is slow. One far behind is sent a snapshot.
Heal queue failing: writes reached a quorum of metadata copies while meta was unreachable; gateways retry queuing them for 10 minutes. Persistent failures mean meta is down: see above.

Repair not running¶

iolake_meta_repair_last_pass_timestamp_seconds hasn't moved in three repair intervals (--repair-interval-secs, an hour by default). Repair runs on the meta leader: - check that there is one (above); - check the leader's log for repair: lines; - a pass that never ends is usually a target OSD that accepts connections but doesn't answer.

Scrub stale: an OSD hasn't finished a scrub pass in 8 days. Check that OSD's log; a pass on a very full disk at 50 MiB/s (--scrub-rate-mib) can take days.

PG scrub stale: a placement group hasn't been scrubbed in two weeks (twice scrub/every_seconds, 7 days). Placement-group scrubs run from the meta leader; check that there is one, and its log. iolakectl pg scrub <pg_id> scrubs one now.

Corrupt shards¶

Scrub or a read found shards that don't match their checksums. The OSD reports them, and repair rebuilds them from the rest of the stripe. If the count keeps rising on one OSD, its drive is failing: check its SMART metrics (iolake_disk_smart_*) and replace it (above).

Placement groups down, incomplete or undersized¶

Peering, on the meta leader, gives each placement group a state. iolakectl pg list --pool <pool> --state <state> names the PGs, and iolakectl pg get <pg_id> --pool <pool> says why and which members. - Down: fewer members answer than a read needs (k). Its objects can't be read or written. Bring the OSDs back. - Incomplete: fewer than a read quorum of its metadata copies answer, so a newer write may be on those that don't. It serves reads, but nothing is rebuilt. Bring the members back. - Undersized: a position has no usable OSD and none can stand in within the pool's rule (not enough hosts, racks or free space). Add OSDs in the missing domain, or bring the member back. - Not clean for an hour with every OSD up: recovery is stuck or slow. iolakectl pg get shows its last error; WaitTooFull means a member to fill is past its full ratio (see disks filling).

Objects unreadable or unrecoverable¶

Some stripes have fewer than k good shards: more failures at once than the code tolerates. Peering counts them per placement group as unfound (iolake_meta_pg_objects_unfound) while members may still bring them back; objects recorded lost (iolake_meta_lost_objects) have fewer than k shards anywhere. 1. Bring back every OSD that is down but intact (powered off, network). Their shards make the stripes readable again. 2. Don't set OSDs out or wipe disks until this is resolved. 3. What can't be brought back is lost. iolakectl pg get gives the counts per placement group; no tool yet lists the objects.

A slow disk¶

p99 shard fsync over 100 ms. Find the OSD from the per-OSD iolake_osd_disk_seconds. A drive getting slower often fails next: check SMART, and drain it (replacing a disk) if it doesn't recover. A drive without power-loss protection is always slow to fsync.

S3 requests failing with 5xx¶

503s during a fault are expected: clients retry.
Persistent 503s mean meta has no leader, OSDs are unreachable, or a gateway refuses writes for clock skew. See those runbooks.
503 SlowDown is a gateway at its memory budget or its in-flight limit (gateway memory and overload): iolake_gateway_requests_rejected_total says which.
A 500 is a bug: report it with the gateway's log line for the request id.

Space leaking: shard deletes are failing, so freed space stays allocated. Nothing reclaims it later yet (iolake#83). It is usually an OSD that is down. A leak is never a loss.

Audit events dropped¶

Events are spooled on the gateway's disk until delivered. Drops mean: - the spool filled (--audit-spool-max-bytes) because a receiver has been down for long; or - a gateway pod was deleted without a clean shutdown.

Fix the receiver; size the spool for the longest outage you accept (audit).

Replication lagging¶

The oldest unreplicated version is over an hour old. That is how much a loss of this site would lose. Check the target's reachability and credentials, and the result="error" count of iolake_replication_sent_total (replication).

Block volumes¶

The block gateway journals writes on its own disk before storing them as chunks. - Slow journal fsync: slows every block write. - Flush failing: chunks aren't being stored; the write cache fills and writes stall. - Read errors: reads that couldn't be served.

The journal is a single copy on the block gateway's disk: losing that disk loses writes not yet flushed (a known gap; see the block design).

Journal failed a write or fsync ("refusing every write until it is reopened" in the gateway's log): every block write fails. Restart the gateway; if it happens again, replace its disk.
Gateway won't start, journal damaged ("damage, not a torn write"): the entries after the damage are acknowledged writes, possibly not yet stored on the OSDs, and this file is their only copy. Keep the file; moving it aside starts the gateway with those writes lost.

Certificates expiring¶

cert-manager renews 30 days before expiry. The services read their certificates only at start, so after a renewal restart them (rolling): kubectl rollout restart statefulset/<name>-meta statefulset/<name>-osd deployment/<name>-gateway, where <name> is the release name if it contains iolake (release iolake: iolake-meta), else <release>-iolake. An expired certificate stops every internal call. There is no expiry metric yet: alert on cert-manager's certmanager_certificate_expiration_timestamp_seconds.

Upgrades¶

Follow upgrades: meta first, then OSDs, then gateways, one at a time. Then iolakectl upgrade status, and iolakectl upgrade finalize once every node runs the new release. Back up a stopped follower's data directory (meta.redb, raft-log) first.