Metrics reference¶
Every Prometheus metric IOLake exports: where it comes from, what it means, and the alerts worth setting on it. For dashboards, Prometheus setup and the console's views of these metrics, see Monitoring.
All metrics are plain Prometheus text exposition. Names start with
iolake_ (and process_ for the standard process family).
Where to scrape¶
| Process | Endpoint | Notes |
|---|---|---|
| Gateway | :9000/metrics |
Or the --admin-listen address when the admin surface is split off. |
| Meta | :9101/metrics |
--metrics-port; 0 serves none. |
| OSD | :9201/metrics |
--metrics-port; 0 serves none. |
| Block gateway | :9301/metrics |
--metrics-listen; empty serves none. |
One gateway (and aio): scrape the gateway alone. By default the
gateway polls every OSD and meta node every 30 seconds and re-exports
their metrics on its own /metrics. Re-exported OSD series carry an
osd_id label, and meta series carry component="meta". The all-in-one
binary runs meta, the OSDs and the block gateway in one process with
their own ports off, so the gateway's /metrics is the only endpoint and
carries everything.
Several gateways (the helm chart): scrape every process directly. Run
gateways with --no-reexport-metrics (the chart does, via
gateway.reexportMetrics: false). Otherwise each gateway exports a copy
of every OSD's and meta's metrics, and every sum multiplies by the number
of gateways. The chart's Prometheus scrapes meta on 9101 and the OSDs on
9201 directly.
Do not do both. Scraping a re-exporting gateway and the OSDs and meta directly counts every OSD and meta series twice.
Series the gateway derives from its own poll (iolake_cluster_*,
iolake_osd_capacity_bytes, the usage and data-safety families) exist
once per gateway. With several gateways, aggregate them with max, not
sum.
Conventions¶
- Latency is a histogram, in seconds, with buckets from 1 ms to 10 s
(or wider where noted). Take percentiles with
histogram_quantile(0.99, sum by (le) (rate(<name>_bucket[5m]))), and keeplein anysum by. - Counters reset when a process restarts. Use
rate()orincrease(), which handle resets. - A family can be absent. Counter and histogram families appear once
something has been recorded. SMART families need
smartctlon the host. Iceberg, Unity, lifecycle and drain families need those features in use. Absent is not the same as zero. - Label cardinality is bounded. Operations, methods, codes and reasons
come from fixed sets. Per-bucket usage (
iolake_bucket_*) is one series per bucket; leave it out with--no-bucket-metricspast tens of thousands of buckets. Per-volume and per-snapshot series are one per volume and snapshot. - gRPC status codes use gRPC's names (
OK,NOT_FOUND,UNAVAILABLE…). A streaming call that fails part-way counts asOK; its status arrives after the response headers, where it is measured.
Metrics¶
S3 requests¶
Exported by: gateway.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_auth_failures_total |
counter | reason |
Requests refused by authentication or authorization (unknown_key, expired, missing_credentials, malformed, denied, signature, clock_skew, internal, policy) |
iolake_iceberg_request_duration_seconds |
histogram | operation |
Iceberg REST catalog request latency |
iolake_iceberg_requests_total |
counter | operation, status |
Iceberg REST catalog requests |
iolake_s3_request_bytes_total |
counter | operation |
Total request body bytes |
iolake_s3_request_duration_seconds |
histogram | operation |
S3 request duration histogram |
iolake_s3_request_errors_total |
counter | code, error, operation |
Failed S3 requests by HTTP code and S3 error name |
iolake_s3_request_size_bytes |
histogram | operation |
Request body size (Content-Length) by operation |
iolake_s3_requests_in_flight |
gauge | operation |
S3 requests currently being served |
iolake_s3_requests_total |
counter | operation, status |
Total S3 requests by operation and status |
iolake_s3_response_bytes_total |
counter | operation |
Total response body bytes |
iolake_unity_request_duration_seconds |
histogram | operation |
Unity Catalog request latency |
iolake_unity_requests_total |
counter | operation, status |
Unity Catalog requests |
Gateway data path¶
Exported by: gateway.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_dedup_dryrun_bytes_total |
counter | bucket, chunking, result |
Bytes the dedup dry-run fingerprinted, and whether already stored |
iolake_dedup_dryrun_chunks_total |
counter | bucket, chunking, result |
Chunks the dedup dry-run fingerprinted, and whether already stored |
iolake_dedup_dryrun_dropped_total |
counter | reason |
Dry-run work given up on (queue full, lookup failed) |
iolake_dedup_dryrun_skipped_total |
counter | reason |
Writes in dry-run buckets that dry-run does not look at (encrypted, inline) |
iolake_erasure_decode_seconds |
histogram | path |
Time to decode one stripe; path=reconstruct means parity was needed |
iolake_erasure_encode_seconds |
histogram | — | Time to erasure-encode one stripe |
iolake_gateway_copied_bytes_total |
counter | mode |
Bytes CopyObject copied, by mode; reference copies move none of them |
iolake_gateway_copies_total |
counter | mode |
CopyObject requests completed, by mode: reference (shares the source's stripes) or bytes (rewritten) |
iolake_gateway_get_aborted_total |
counter | reason |
GETs cut off after their headers were sent: read (a stripe could not be read), length (a stripe read back other than its recorded size), memory (no room in the budget for the next stripe in time) |
iolake_gateway_inline_bytes_total |
counter | — | Bytes of objects stored in their metadata |
iolake_gateway_inline_objects_total |
counter | — | Objects small enough to be stored in their metadata, with no shards |
iolake_gateway_memory_budget_bytes |
gauge | — | The memory budget for request bodies and GET read-ahead (--memory-budget) |
iolake_gateway_memory_budget_in_use_bytes |
gauge | — | Bytes of the memory budget held by requests now |
iolake_gateway_osd_request_errors_total |
counter | address, kind |
Failed shard calls to an OSD (timeout, refused, error) |
iolake_gateway_protection_data_shards |
gauge | scheme |
Number of data shards (k) |
iolake_gateway_protection_efficiency |
gauge | scheme |
Storage efficiency ratio (data/total) |
iolake_gateway_protection_lrc_global_parity |
gauge | — | LRC global parity shards (LRC pools only) |
iolake_gateway_protection_lrc_local_parity |
gauge | — | LRC local parity shards (LRC pools only) |
iolake_gateway_protection_parity_shards |
gauge | scheme |
Number of parity shards (m) |
iolake_gateway_protection_total_shards |
gauge | scheme |
Total shards (data + parity) |
iolake_gateway_rdma_fallbacks_total |
counter | direction, reason |
Shards sent over gRPC to an OSD that offers Transfer Engine |
iolake_gateway_request_phase_seconds |
histogram | operation, phase |
Time spent in each phase of a request, by operation and phase |
iolake_gateway_requests_admitted |
gauge | — | Requests admitted and not yet finished, response bodies included |
iolake_gateway_requests_limit |
gauge | — | Requests the gateway serves at once (--max-requests) |
iolake_gateway_requests_rejected_total |
counter | reason |
Requests refused with 503 SlowDown, by the limit they hit (requests or memory) |
iolake_gateway_shard_checksum_mismatches_total |
counter | direction |
Shards sent over gRPC that did not match their checksum |
iolake_gateway_shard_latency_seconds |
histogram | address, direction |
Time for one shard read or write to an OSD, by OSD |
iolake_gateway_shard_reclaim_failures_total |
counter | reason |
Shard deletes that failed, leaving the block allocated |
iolake_gateway_shard_transfers_total |
counter | direction, transport |
Shards moved to or from OSDs, by direction and transport (rdma or grpc) |
iolake_gateway_shards_reclaimed_total |
counter | reason |
Shards deleted because nothing references them any more |
iolake_gateway_timeouts_total |
counter | kind |
Requests or connections cut off by a timeout: body (a request body stopped arriving), send (a client stopped reading) |
iolake_gateway_uptime_seconds |
counter | — | Gateway uptime in seconds |
iolake_read_locality_bytes_total |
counter | locality |
Bytes fetched from OSDs, by topology distance |
Capacity and usage¶
Exported by: gateway (poll).
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_bucket_last_modified_timestamp_seconds |
gauge | bucket, tenant |
Newest write or delete in the bucket |
iolake_bucket_logical_bytes |
gauge | bucket, tenant |
Bytes of current objects as uploaded |
iolake_bucket_noncurrent_bytes |
gauge | bucket, tenant |
Bytes of noncurrent versions as uploaded |
iolake_bucket_noncurrent_versions |
gauge | bucket, tenant |
Noncurrent object versions |
iolake_bucket_objects |
gauge | bucket, tenant |
Current objects |
iolake_bucket_quota_bytes |
gauge | bucket, tenant |
Storage quota (0 = unlimited) |
iolake_s3_quota_refusals_total |
counter | scope (bucket, tenant) |
Writes refused for passing a quota (403 QuotaExceeded) |
iolake_bucket_stored_bytes |
gauge | bucket, tenant |
Bytes on disk for all versions incl. parity |
iolake_capacity_poll_duration_seconds |
gauge | — | How long the last poll took |
iolake_capacity_poll_failures_total |
counter | — | Polls in which meta did not answer |
iolake_capacity_poll_last_success_timestamp_seconds |
gauge | — | When the gateway last refreshed capacity, usage and cluster metrics |
iolake_capacity_polls_total |
counter | — | Polls attempted |
iolake_cluster_available_bytes |
gauge | — | Raw bytes free across OSDs that answered the last poll |
iolake_cluster_buckets |
gauge | — | Buckets |
iolake_cluster_capacity_bytes |
gauge | — | Raw capacity across OSDs that answered the last poll |
iolake_cluster_logical_bytes |
gauge | — | Bytes of current objects as uploaded, all buckets |
iolake_cluster_objects |
gauge | — | Current objects across all buckets |
iolake_cluster_osds_total |
gauge | — | OSDs registered with the cluster |
iolake_cluster_osds_up |
gauge | — | OSDs that answered the last poll |
iolake_cluster_stored_bytes |
gauge | — | Bytes on disk for all object versions incl. parity |
iolake_cluster_usable_capacity_bytes |
gauge | — | Raw capacity scaled by the default protection efficiency |
iolake_cluster_used_bytes |
gauge | — | Raw bytes allocated across OSDs that answered the last poll |
iolake_osd_capacity_bytes |
gauge | address, osd_id |
Raw capacity of one OSD |
iolake_osd_shards |
gauge | address, osd_id |
Shards stored on one OSD |
iolake_osd_up |
gauge | address, osd_id |
1 if the OSD answered the last poll |
iolake_osd_uptime_seconds |
counter | osd_id |
OSD uptime in seconds |
iolake_osd_used_bytes |
gauge | address, osd_id |
Raw bytes allocated on one OSD |
iolake_tenant_buckets |
gauge | tenant |
Buckets owned by the tenant |
iolake_tenant_last_modified_timestamp_seconds |
gauge | tenant |
Newest write or delete in any of the tenant's buckets |
iolake_tenant_logical_bytes |
gauge | tenant |
Bytes of current objects as uploaded |
iolake_tenant_objects |
gauge | tenant |
Current objects |
iolake_tenant_quota_bytes |
gauge | tenant |
Storage quota (0 = unlimited) |
iolake_tenant_stored_bytes |
gauge | tenant |
Bytes on disk for all versions incl. parity |
iolake_usage_last_update_timestamp_seconds |
gauge | — | When usage was last refreshed |
iolake_usage_osds_stale |
gauge | — | OSDs whose usage is carried over from an earlier poll |
Data safety, drain and rebalance¶
Exported by: gateway (poll).
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_at_risk_bytes |
gauge | — | Logical bytes of at-risk objects |
iolake_degraded_bytes |
gauge | — | Logical bytes of degraded objects |
iolake_drain_errored |
gauge | node_id |
1 when a drain has hit errors |
iolake_drain_shards_initial |
gauge | node_id |
Shards a draining OSD held when the drain began |
iolake_drain_shards_migrated_total |
counter | node_id |
Shards moved off a draining OSD |
iolake_drain_shards_remaining |
gauge | node_id |
Shards still to move off a draining OSD |
iolake_objects_at_risk |
gauge | — | Objects with fewer than k+1 reachable shards: one more loss makes them unreadable |
iolake_objects_checked |
gauge | — | Objects and versions whose shard availability was checked |
iolake_objects_degraded |
gauge | — | Objects with a shard on an OSD that is down (still readable) |
iolake_objects_unreadable |
gauge | — | Objects with fewer than k reachable shards |
iolake_rebalance_last_sweep_timestamp_seconds |
gauge | — | When the balancer last ran |
iolake_rebalance_pg_candidates |
gauge | — | Placement groups the balancer wanted to move on its last tick |
iolake_rebalance_pgs_moved_total |
counter | — | Placement groups the balancer has moved |
iolake_rebalance_running |
gauge | — | 1 if the balancer is started and not paused |
iolake_safety_osds_unreported |
gauge | — | OSDs that did not answer, so objects only they hold metadata for were not checked |
iolake_unreadable_bytes |
gauge | — | Logical bytes of unreadable objects |
Lifecycle and multipart¶
Exported by: gateway / meta.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_lifecycle_actions_total |
counter | action, result |
Lifecycle actions (expire, delete_marker, abort_upload) and whether they succeeded |
iolake_lifecycle_scan_seconds |
histogram | — | Time one lifecycle scan took |
iolake_lifecycle_scans_total |
counter | result |
Lifecycle scans over every bucket |
iolake_multipart_upload_oldest_age_seconds |
gauge | — | Age of the oldest open multipart upload |
iolake_multipart_upload_parts_bytes |
gauge | — | Bytes held by parts of open multipart uploads |
iolake_multipart_uploads_older_than |
gauge | age |
Open multipart uploads older than the given age |
iolake_multipart_uploads_open |
gauge | — | Multipart uploads started and neither completed nor aborted |
Block gateway¶
Exported by: block gateway.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_block_cache_clean_bytes |
gauge | — | Bytes of stored chunks held in the cache for reads |
iolake_block_cache_dirty_bytes |
gauge | — | Bytes written and journaled but not yet stored as stripes |
iolake_block_cache_dirty_chunks |
gauge | — | Chunks written and journaled but not yet stored as stripes |
iolake_block_cache_volumes |
gauge | — | Volumes with a cache on this gateway |
iolake_block_chunk_flush_seconds |
histogram | result |
Time to store one chunk: encode, write its shards, record it in meta |
iolake_block_chunk_reads_total |
counter | source |
Chunks a read had to fetch, not in the cache: stored (from their stripe) or unwritten (zeros) |
iolake_block_chunks_flushed_total |
counter | result |
Chunk flushes, by result: stored, conflict (changed meanwhile, retried) or failed (retried) |
iolake_block_io_bytes_total |
counter | protocol, op |
Bytes read or written by block requests that succeeded |
iolake_block_io_errors_total |
counter | protocol, op |
Block requests that failed |
iolake_block_io_seconds |
histogram | protocol, op |
Time to serve one block request, by protocol and operation |
iolake_block_journal_fsync_seconds |
histogram | — | Time to fsync the block write journal; every acknowledged write waits for one |
iolake_block_shard_delete_failures_total |
counter | reason |
Shard deletes that failed while freeing stripes, leaving space allocated |
iolake_block_stripes_freed_total |
counter | reason |
Stripes deleted because nothing uses them any more (overwrite, volume, snapshot) |
Block volumes and snapshots¶
Exported by: meta.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_block_snapshot_chunks |
gauge | snapshot_id, volume_id, name |
Chunks one snapshot holds (shared with its volume until overwritten) |
iolake_block_snapshots_total |
gauge | — | Block snapshots |
iolake_block_volume_size_bytes |
gauge | volume_id, name, pool |
Size of one block volume |
iolake_block_volume_used_bytes |
gauge | volume_id, name, pool |
Chunks one block volume has written, in bytes |
iolake_block_volumes_by_state |
gauge | state |
Block volumes by state |
iolake_block_volumes_provisioned_bytes |
gauge | — | Size of every block volume added up |
iolake_block_volumes_total |
gauge | — | Block volumes |
iolake_block_volumes_used_bytes |
gauge | — | Chunks block volumes have written, in bytes (volumes are thin: unwritten chunks take nothing) |
Meta service¶
Exported by: meta.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_meta_buckets_total |
gauge | — | Total number of buckets |
iolake_meta_commit_seconds |
histogram | — | Time to commit one redb write transaction (includes fsync) |
iolake_meta_grpc_latency_seconds |
histogram | method |
Time meta took to answer a gRPC call, by method |
iolake_meta_grpc_requests_total |
counter | code, component, method |
gRPC calls meta served, by method and status code |
iolake_meta_osds_total |
gauge | — | Total registered OSDs |
iolake_meta_shared_stripe_referrers |
gauge | — | References held on shared stripes; a stripe is freed when its last one goes |
iolake_meta_shared_stripes |
gauge | — | Stripes used by more than one object, chunk, snapshot or clone (zero-copy copies, snapshots) |
iolake_meta_uptime_seconds |
counter | — | Metadata service uptime |
iolake_meta_users_total |
gauge | — | Total users |
Meta Raft¶
Exported by: meta.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_meta_raft_applied_index |
gauge | — | Index of the last entry applied to this node's state |
iolake_meta_raft_has_leader |
gauge | — | 1 while this node knows a leader; 0 means no writes can commit |
iolake_meta_raft_is_leader |
gauge | — | 1 on the Raft leader, 0 elsewhere |
iolake_meta_raft_last_log_index |
gauge | — | Index of the last entry in this node's Raft log |
iolake_meta_raft_purged_index |
gauge | — | Index up to which the log has been compacted away |
iolake_meta_raft_replication_lag_entries |
gauge | peer |
Entries a follower is behind the leader (leader only; multi-node meta) |
iolake_meta_raft_snapshot_build_seconds |
histogram | — | Time to build a Raft snapshot of the whole state machine (built in memory) |
iolake_meta_raft_snapshot_bytes |
gauge | — | Size of the last Raft snapshot built (held in memory while sent) |
iolake_meta_raft_snapshot_index |
gauge | — | Index the last snapshot covers |
iolake_meta_raft_snapshot_install_seconds |
histogram | — | Time to install a Raft snapshot received from the leader |
iolake_meta_raft_term |
gauge | — | Current Raft term; a rising term means elections |
iolake_meta_raft_voters |
gauge | — | Voting members of the Raft cluster |
Meta disk¶
Exported by: meta, each node. See meta's disk filling.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_meta_disk_free_bytes |
gauge | — | Bytes free on the filesystem of meta's data directory; -1 when it can't be told |
iolake_meta_disk_reserve_bytes |
gauge | — | The reserve meta keeps free (--min-free-mib): under it the leader refuses writes |
iolake_meta_disk_full |
gauge | — | 1 while the node is under its reserve, or a write of its own found the filesystem full in the last 10 s |
iolake_meta_writes_refused_no_room_total |
counter | — | Writes this node refused, as leader, for want of room (507 to S3 clients) |
iolake_meta_disk_out_of_space_total |
counter | — | Writes to the Raft log or a snapshot that found the filesystem full (ENOSPC, EDQUOT) and waited |
Repair¶
Exported by: meta (leader).
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_meta_repair_block_stripes_checked_total |
counter | — | Block chunk stripes checked by the repairer (also counted in objects checked) |
iolake_meta_repair_errors_total |
counter | — | Repairs that failed and will be retried next pass |
iolake_meta_repair_last_pass_seconds |
gauge | — | How long the last completed repair pass took (on the node that ran it) |
iolake_meta_repair_last_pass_timestamp_seconds |
gauge | — | When the last repair pass completed (Unix time; 0: none yet on this node) |
iolake_meta_repair_objects_checked_total |
counter | — | Objects checked by the repairer |
iolake_meta_repair_passes_total |
counter | — | Completed repair passes |
iolake_meta_repair_shards_rebuilt_total |
counter | reason |
Shards rebuilt from the rest of their stripe |
iolake_meta_repair_unrecoverable_stripes_total |
counter | — | Stripes found with fewer than k good shards |
Placement groups¶
Exported by: meta; the leader counts them, so take the max over meta
nodes. The states are peering's (Down, Incomplete,
Undersized, Degraded, Clean) and recovery's while it works a PG
(Recovering, Backfilling, WaitTooFull); iolakectl pg list and
iolakectl pg get show the same per PG.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_meta_pgs |
gauge | state |
Placement groups by the state peering last found |
iolake_meta_pgs_not_clean |
gauge | — | Placement groups in any state but Clean |
iolake_meta_pg_not_clean_oldest_seconds |
gauge | — | How long the PG longest out of Clean has been |
iolake_meta_pgs_undersized |
gauge | — | PGs with a member no OSD can stand in for within the pool's rule |
iolake_meta_osds_down |
gauge | — | OSDs in service (in) that aren't up |
iolake_meta_pg_objects_degraded |
gauge | — | Objects some acting member of their PG lacks something of |
iolake_meta_pg_objects_unfound |
gauge | — | Objects with fewer complete copies than a read needs among the members that answered |
iolake_meta_pg_copies_missing |
gauge | — | Metadata copies acting members lack |
iolake_meta_pg_shards_missing |
gauge | — | Shards acting members should hold at their position and don't |
iolake_meta_lost_objects |
gauge | — | Objects recorded lost: fewer than k shards left. Above 0 is data loss |
iolake_meta_pg_peered_total |
counter | — | Placement groups peered |
iolake_meta_pg_listed_total |
counter | — | PGs whose members disagreed by summary, so were listed |
iolake_meta_pg_listings_added_total |
counter | — | Listing entries peering put back |
iolake_meta_pg_listings_removed_total |
counter | — | Listing entries peering removed (their key was deleted) |
iolake_meta_pg_recovering |
gauge | — | PGs being recovered |
iolake_meta_pg_recovery_backing_off |
gauge | — | PGs whose recovery is waiting out a backoff |
iolake_meta_pg_recovery_objects_remaining |
gauge | — | Objects recovery has left in the PGs it is working |
iolake_meta_pg_recovery_bytes_remaining |
gauge | — | Bytes of those objects |
iolake_meta_pg_reservations |
gauge | — | Recovery reservations held, as (OSD, PG) pairs |
iolake_meta_pg_reservations_refused_total |
counter | — | Times a PG couldn't reserve a slot on every OSD it needed |
iolake_meta_pg_recovery_retries_total |
counter | — | Recovery passes that got nowhere (each waits twice as long, to 5 min) |
iolake_meta_pg_too_full_total |
counter | — | Times a PG waited for a member that was too full |
iolake_meta_pg_recovered_objects_total |
counter | — | Objects recovery made whole |
iolake_meta_pg_recovered_shards_total |
counter | — | Shards recovery wrote to the member at their position |
iolake_meta_pg_recovered_copies_total |
counter | — | Metadata copies recovery wrote |
iolake_meta_pg_replica_copies_total |
counter | — | Replicated copies recovery made from an intact one |
iolake_meta_pg_remaps_total |
counter | — | PGs moved to their up set |
iolake_meta_pg_lrc_rebuilds_total |
counter | kind |
LRC shards rebuilt, from their local group or the global stripe |
iolake_meta_pg_lrc_shards_read_total |
counter | kind |
Shards those rebuilds read |
iolake_meta_scrub_pgs_completed_total |
counter | — | Placement-group scrubs completed |
iolake_meta_scrub_shards_total |
counter | — | Shards checked by PG scrubs |
iolake_meta_scrub_bytes_total |
counter | — | Shard bytes read back and checked by PG scrubs |
iolake_meta_scrub_shards_bad_total |
counter | reason |
Shards a scrub found missing or corrupt (recovery rebuilds them) |
iolake_meta_scrub_errors_total |
counter | — | Scrub steps a member couldn't take; retried |
iolake_meta_scrub_pgs_never |
gauge | — | PGs not yet scrubbed |
iolake_meta_scrub_oldest_seconds |
gauge | — | Age of the oldest completed PG scrub |
OSD¶
Exported by: osd.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_osd_corrupt_shards |
gauge | — | Corrupt shards on this OSD not yet rebuilt |
iolake_osd_corrupt_shards_found_total |
counter | — | Shards found failing their checksum, by the scrubber or a read |
iolake_osd_disk_seconds |
histogram | op, |
Time of one shard disk operation: write, sync (before the write is acknowledged), read |
iolake_osd_grpc_bytes_received_total |
counter | method, |
Bytes of shard requests received, by method |
iolake_osd_grpc_bytes_sent_total |
counter | method, |
Bytes of shard responses sent, by method |
iolake_osd_grpc_latency_seconds |
histogram | method, |
Time this OSD took to answer a gRPC call, by method |
iolake_osd_grpc_requests_total |
counter | code, method, |
gRPC calls this OSD served, by method and status code |
iolake_osd_info |
gauge | datacenter, node, rack, region |
OSD information |
iolake_osd_scrub_bytes_total |
counter | — | Bytes read and checked by the scrubber |
iolake_osd_scrub_last_pass_seconds |
gauge | — | How long the last completed scrub pass took |
iolake_osd_scrub_last_pass_timestamp_seconds |
gauge | — | When the last scrub pass completed (Unix time; 0: none yet) |
iolake_osd_scrub_passes_total |
counter | — | Completed scrub passes |
iolake_osd_scrub_shards_total |
counter | — | Shards read and checked by the scrubber |
iolake_osd_shards_total |
gauge | — | Total shards stored |
iolake_osd_wal_fsync_seconds |
histogram | — | Time for one metadata WAL fdatasync |
iolake_osd_wal_records_synced_total |
counter | — | Metadata WAL records made durable; divide by syncs for records per fsync |
iolake_osd_wal_syncs_total |
counter | — | Metadata WAL fdatasyncs |
Disks¶
Exported by: osd.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_disk_available_bytes |
gauge | disk, |
Free bytes on the disk |
iolake_disk_capacity_bytes |
gauge | disk, |
Disk capacity |
iolake_disk_errors_total |
counter | disk, type |
read: failed reads; write: failed writes and flushes; checksum: checksum mismatches |
iolake_disk_healthy |
gauge | disk, |
Disk health status (1=healthy, 0=unhealthy) |
iolake_disk_read_bytes_total |
counter | disk, |
Bytes read |
iolake_disk_reads_total |
counter | disk, |
Shard reads |
iolake_disk_shards_total |
gauge | disk, |
Shards on disk |
iolake_disk_smart_check_errors_total |
counter | — | SMART checks that failed (no smartctl, unsupported device) |
iolake_disk_smart_checks_total |
counter | — | SMART checks run |
iolake_disk_smart_health_score |
gauge | device |
SMART-derived health score |
iolake_disk_smart_healthy |
gauge | device |
SMART overall health (1 healthy); only where smartctl is installed |
iolake_disk_smart_pending_sectors |
gauge | device |
Sectors pending reallocation |
iolake_disk_smart_power_on_hours |
counter | device |
Drive power-on hours |
iolake_disk_smart_predicted_failure_days |
gauge | device |
Predicted days to failure, when SMART data allows a prediction |
iolake_disk_smart_reallocated_sectors |
gauge | device |
Reallocated sectors: a rising count predicts failure |
iolake_disk_smart_temperature_celsius |
gauge | device |
Drive temperature |
iolake_disk_used_bytes |
gauge | disk, |
Disk used space |
iolake_disk_writes_total |
counter | disk, |
Shard writes |
iolake_disk_written_bytes_total |
counter | disk, |
Bytes written |
Process¶
Exported by: every process.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
iolake_build_info |
gauge | commit, version |
Build version and commit; always 1 |
process_cpu_seconds_total |
counter | — | User and system CPU time |
process_max_fds |
gauge | — | File descriptor limit |
process_open_fds |
gauge | — | Open file descriptors |
process_start_time_seconds |
gauge | — | Process start, seconds since the epoch |
Alerts¶
The chart ships the alert rules (deploy/helm/iolake/alerts/iolake-rules.yaml
in the code repo). Production lists them, with a
runbook for each.
Known gaps¶
- Gateway to meta calls are measured only on meta's side
(
iolake_meta_grpc_latency_seconds), which leaves out network time. - Lifecycle expiry deletes an object's metadata but not its shards
(a known leak);
iolake_lifecycle_actions_total{action="expire"}counts the deletes, and the space they leave is not reclaimed yet. - The Transfer Engine (RDMA) path reports transfers and fallbacks, but not its own latency.
- Tracing (OpenTelemetry spans across gateway, meta and OSD) is not implemented.