Monitoring

How to watch an IOLake cluster: where its metrics are, how to have Prometheus collect them, the dashboards and console views built on them, and the health endpoints. Every metric, its labels and the alerts worth setting on it are in the metrics reference.

What exports metrics

Process	Metrics endpoint	Health
Gateway	:9000/metrics (or --admin-listen)	/health, /_ready
Meta	:9101/metrics (--metrics-port, 0 = off)	:9101/health
OSD	:9201/metrics (--metrics-port, 0 = off)	:9201/health
Block gateway	:9301/metrics (--metrics-listen, empty = off)	:9301/health

By default the gateway also re-exports the OSDs' and meta's metrics, which it polls every 30 seconds. That way one scrape target covers a small cluster: aio, or a single gateway. With several gateways that duplicates every series, so turn it off with --no-reexport-metrics and scrape each process directly. The metrics reference explains the choice.

aio

iolake-aio runs meta, the OSDs, the gateway and (with --block-port) the block gateway in one process. Their own metrics ports are off and everything is on the gateway's /metrics:

curl -s http://localhost:9000/metrics | grep iolake_meta_raft_is_leader

A Prometheus scrape config for it:

scrape_configs:
  - job_name: iolake
    static_configs:
      - targets: ['localhost:9000']

Helm

The chart runs gateways with --no-reexport-metrics (gateway.reexportMetrics: false). With monitoring.enabled, it deploys a Prometheus that discovers and scrapes the gateway service (:9000), each meta pod (:9101) and each OSD pod (:9201). For a Prometheus of your own, scrape those three the same way: every gateway pod, every meta pod, every OSD pod.

Turn gateway.reexportMetrics on only for a single gateway scraped on its own.

Prometheus and the console

The console (/_console/) charts these metrics under Monitoring:

Tab	Shows
Requests	S3 request rate, latency percentiles, errors, Iceberg operations
Block storage	block operations, latency, throughput and errors by protocol; unflushed data; journal fsync; chunk flushes
Metadata	Raft leader, term, applied index and replication lag; meta call rate, latency and errors; durable commit time
Data protection	degraded, at-risk and unreadable objects; when repair and scrub last ran; shards rebuilt; corrupt shards; scrub throughput
Disks & OSDs	OSDs up, raw capacity used; disk write, sync and read latency; per-OSD shard write latency; WAL fsync; OSD errors

Without Prometheus, the console reads the gateway's /metrics every five seconds. It computes rates and percentiles from what changed between reads, and keeps the last five minutes. With --prometheus-url set on the gateway, the longer windows (1 h to 7 d) come from Prometheus range queries. Under --no-reexport-metrics, the OSD and meta views need Prometheus, since the gateway's /metrics then no longer carries them.

Grafana dashboards

deploy/monitoring/grafana/dashboards/ (and the same set in deploy/helm/iolake/dashboards/) has:

Dashboard	Covers
cluster-overview.json	capacity, objects, request rate and latency, disk usage
s3-operations.json	S3 requests by operation, latency, errors, requests in flight
osd-detail.json	per-OSD capacity, disk health, gRPC request rate, latency (p99) and errors
disk-health.json	SMART readings (needs smartctl on the OSD hosts)
block-storage.json	block volumes, provisioned vs used, block operations, throughput, snapshots

deploy/monitoring/ also has a docker-compose Prometheus and Grafana that load these dashboards.

Alerting

The chart ships a rule file (deploy/helm/iolake/alerts/iolake-rules.yaml in the code repo); production lists every alert with its runbook. It covers: - data safety: objects unreadable, at risk, unfound or lost, repair not running, scrub stale, corrupt shards; - placement groups: down, incomplete, undersized, not clean for an hour with every OSD up, not scrubbed in two weeks; - metadata: no Raft leader, repeated elections, a follower falling behind, the heal queue failing, a gateway's clock skewed; - capacity and OSDs down; - durability latency; - S3 server errors, and space leaking from failed shard deletes; - audit events dropped, replication lagging; - block journal latency, flushes failing, read errors.

Data-safety alerts come first. iolake_objects_unreadable above zero, or iolake_meta_raft_has_leader at zero, needs someone now.

Health endpoints

• GET /health (gateway) answers 200 {"status":"healthy"} while the process is up. It checks nothing else, so use it for liveness.
• GET /_ready (gateway) answers 503 until meta answers and the OSDs are reachable, then 200 from then on. If some OSDs are still missing after a grace period, it becomes ready as long as enough OSDs for a write quorum are reachable. It is a startup gate for load balancers and readiness probes. It does not drop when OSDs fail later; that is what iolake_cluster_osds_up and the data-safety metrics are for. It answers 503 again ("status": "stopping") once the gateway is shutting down, while it still serves (see how a node stops).
• GET /health on an OSD's metrics port answers 200 while every one of its disks is healthy, and 503 (UNHEALTHY) once one has had a failed read, write or flush, or a checksum mismatch, since the OSD started (iolake_disk_errors_total, iolake_disk_healthy).
• GET /health on the block gateway's metrics port answers 200 while the process is up.
• GET /health on meta's metrics port and on its admin port (9102) answers 200 while the process is up and its Raft runs, and 503 (with the reason) once its Raft has stopped for good, after a storage error. Meta also exits then, so its supervisor restarts it. GET /ready on the admin port answers 200 once the node is a member and knows a leader. The Helm chart's probes use the admin port's two. GET /status on the admin port shows the reason as fatal.

Logs

Every process logs to stderr in plain text. Set the level with --log-level (error, warn, info, debug, trace; default info). Things worth finding in logs that metrics only count: - repair work: repair: rebuilt … and repair: … needs k; - scrub results: scrub pass done: … corrupt; - shard reclaim failures, which leave space allocated; - gateway readiness, including ready without every OSD.

Tracing

Not implemented. Requests are not traced across gateway, meta and OSD; per-phase timing of S3 PUTs and GETs is in iolake_gateway_request_phase_seconds.