Skip to content

Operations Guide

Day-to-day operations, monitoring, and maintenance of IOLake clusters.

Contents

Quick Reference

Service Management

# Start all services
sudo systemctl start iolake-gateway iolake-meta iolake-osd

# Stop all services
sudo systemctl stop iolake-gateway iolake-meta iolake-osd

# Restart a specific service
sudo systemctl restart iolake-gateway

# Check service status
sudo systemctl status iolake-gateway

Health Checks

# Gateway: process up, and ready to serve (meta and OSDs reachable)
curl http://localhost:9000/health
curl http://localhost:9000/_ready

# Meta: this node's view of the Raft group (role, leader, term, voters)
curl -s http://meta1:9102/status

# OSDs: status and admin state of each
iolakectl node list

# Placement groups not Clean, and every node's release
iolakectl pg list --state Degraded
iolakectl upgrade status

Common Operations

Task Command
List buckets iolakectl bucket list
Check capacity and usage iolakectl cluster usage, iolakectl node show <osd-id>
View cluster map iolakectl cluster topology
Check data safety curl -s http://<gateway>:9000/metrics \| grep iolake_objects_at_risk, or iolakectl metrics query 'max(iolake_objects_at_risk)' with Prometheus (below); each OSD counts at most once a minute, and at once when a node goes up or down
Drain OSD iolakectl osd set-state <osd-id> draining, then iolakectl cluster drain-status

iolakectl metrics query runs PromQL through the gateway, which forwards it to the Prometheus named by its --prometheus-url; without one it answers that none is configured. Reading /metrics (on the admin listener, if split off) needs no Prometheus.

Cluster States

No single command prints a cluster state; these signals make it up:

State Signals
Healthy Every OSD up and in (iolakectl node list); meta has a leader (iolake_meta_raft_has_leader 1); every placement group Clean (iolake_meta_pgs_not_clean 0); iolake_objects_at_risk 0
Degraded An OSD down or PGs Degraded, Recovering or Backfilling: data readable, redundancy being restored. A down OSD's PGs get a stand-in after 120 s (pg/down_out_seconds) and backfill onto it
Critical iolake_objects_unreadable above 0, PGs Down or Incomplete, or meta without a leader: bring failed OSDs or meta nodes back (runbooks)

Maintenance Windows

Rolling Upgrades

Upgrades has the procedure: back up meta, then the meta nodes (followers first, leader last), then the OSDs one failure domain at a time, then the gateways, then iolakectl upgrade finalize. In short:

# 1. Meta, one node at a time: which one leads?
curl -s http://meta1:9102/status
sudo systemctl restart iolake-meta      # on meta2, meta3, then meta1

# 2. OSDs, one host at a time. No drain: a restart doesn't need one.
sudo systemctl restart iolake-osd
iolakectl node list                          # up again before the next

# 3. Gateways
sudo systemctl restart iolake-gateway

# 4. Every node on the new release, then finalize
iolakectl upgrade status
iolakectl upgrade finalize

Adding Capacity

Start an OSD on the new host; it registers itself with meta and joins placement. Placement groups rebalance onto it in the background:

iolakectl node list                 # the new OSD, "in"
iolakectl cluster rebalance status

Removing an OSD

iolakectl osd set-state <osd-id> draining     # copies everything off it
iolakectl cluster drain-status                # until it is "out"

When the drain finishes the OSD is marked out and wiped. Then stop it and take the hardware away.

Backup and Recovery

Metadata Backup

Each meta node keeps its state in <data-dir>/meta.redb and its Raft log beside it. The log is compacted into a snapshot every --raft-snapshot-every entries (default 500,000), keeping the last --raft-keep-logs (10,000), so it doesn't grow without bound (meta HA).

To back up, stop one follower and copy its data directory (or snapshot its volume: on Kubernetes, the meta pod's PVC with your storage class's snapshot mechanism), then start it again. There is no online export command. A fresh install against empty volumes starts a new cluster with no history: restore the volumes first.

Object Recovery

Repair is continuous: any stripe with at least k readable shards is rebuilt automatically. An object below k shards can't be recovered by any command; iolake_meta_repair_unrecoverable_stripes_total counts them.

Troubleshooting Quick Reference

Symptom Check Solution
Slow uploads Network, OSD load Scale gateways, add OSDs
High latency Metadata service Check Raft leader via curl meta-N:9102/status, disk I/O
Objects missing OSD health iolake_objects_at_risk, iolake_meta_repair_unrecoverable_stripes_total
Auth failures User credentials Verify access keys
Connection refused Service status Restart services

Emergency Procedures

Complete Cluster Recovery

If all metadata nodes fail but their PVCs survive:

  1. Restart the meta nodes (on Kubernetes, kubectl rollout restart statefulset/<name>-meta, where <name> is the release name if it contains iolake, else <release>-iolake): each reads its persisted Raft log and state, and the group elects a leader once a majority is up.
  2. OSDs + gateways keep running throughout.

If the PVCs are lost (unrecoverable storage-layer failure):

  1. Restore PVCs from the latest snapshot if you have one.
  2. Otherwise the cluster's metadata is lost: delete the meta StatefulSet and its PVCs, then helm upgrade — the bootstrap Job forms a fresh Raft group. Tenants, users, buckets and the objects' metadata are gone with it.

Data Recovery from Raw Disks

Not available: object metadata lives on the OSDs and meta's Raft log, but there is no tool yet to rebuild a cluster's metadata from raw disks. Keep meta's PVCs snapshotted (above).

See Failure Recovery for detailed recovery procedures.