Skip to content

Meta high availability

Meta runs as a Raft group, normally three nodes. A group of three keeps serving while any two are up; five, while any three are.

What clients need

Give every gateway, OSD and block gateway every meta node's address, comma-separated:

--meta-endpoint http://meta1:9100,http://meta2:9100,http://meta3:9100

In Kubernetes, the chart's meta Service does the same job: point clients at it.

  • Any meta node serves any call. A follower forwards the call to the Raft leader and relays the answer, so clients never see a follower's possibly stale state, and never need to know which node leads.
  • A node that stops answering is skipped. Clients probe each meta address every second, with a one-second timeout, and leave out the ones that don't answer until they do. That covers a node that is down, and one that is frozen or cut off but still accepting connections, and a leader that says it can't serve (below).
  • The leader answers only while a majority confirms it leads. A leader cut off from the other meta nodes still believes it leads, but it stops answering within about 3 seconds (calls fail, 503 to S3 clients) rather than serve state the others may have changed since: a key revoked on the majority's side is not accepted on its side.
  • No call waits forever. A call to meta gives up after 5 seconds without an answer, and a connection to a meta node that stops answering part way through one is closed within about 5 seconds, failing the call. The gateway answers S3 clients 503 when meta can't be reached, which they retry.

What happens when a meta node is lost

Lost Effect
A follower Nothing visible. Calls in flight to it fail and are retried; new calls go to the others. When it returns it catches up from the leader.
The leader The others stand for election after hearing nothing from it for 3–4 s (its 2 s lease and a random 1–2 s timeout, --raft-election-ms, 1000; heartbeats every 500 ms), and elect a new one. Writes resume within a few seconds; calls in flight at the moment of loss fail (503) and are retried. No acknowledged write is lost.
The leader, cut off (frozen, partitioned) As above. While cut off it answers nothing from its own state: within about 2 seconds its calls fail (503) and clients go to the others. When it returns it finds a newer term and steps down: it never leads a second cluster.
A majority Writes are refused (503) rather than accepted without a quorum. Service resumes as soon as a majority is back, with nothing acknowledged lost.

What is replicated

Everything clients rely on lives in the Raft log, so a new leader has it: buckets, users and keys (including the bootstrap admin), policies, tenants, pools, placement groups, the shared-stripe registry, packs, block volumes and chunks, OSD registrations, KMS keys, and multipart uploads in progress.

The bootstrap admin is created once for the cluster, by the first leader; every meta node writes the same admin-creds.env.

Log compaction

The Raft log doesn't grow without bound. Every --raft-snapshot-every entries (default 500,000), each meta node writes a snapshot of its state and drops the log up to it, keeping the last --raft-keep-logs entries (default 10,000). A node that falls further behind, or a new node, is sent a snapshot of the whole metadata database. It travels in chunks and goes through <data-dir>/raft-snapshots/, so leave free space there of about the size of meta.redb. /status on the admin port shows how far the last snapshot (snapshot) and the purge (purged) reach.

Replacing a meta node

Use the Raft admin API on the leader (port 9102 by default; with mTLS, HTTPS and a certificate of role admin: curl --cacert ca.pem --cert admin.pem --key admin.key https://meta1:9102/...):

# 1. Start the new node (empty data dir, a new --node-id), then add it.
curl -X POST meta1:9102/add-learner -H 'content-type: application/json' -d '{"node_id": 4, "addr": "meta4:9100"}'
# 2. Make the voters the surviving nodes plus the new one.
curl -X POST meta1:9102/change-membership -H 'content-type: application/json' -d '{"voters": [1, 2, 4]}'
# 3. Stop the old node and remove its data.
curl meta1:9102/status   # voters, leader, and this node's own last applied index

Add the new node's address to clients' --meta-endpoint.

How it's tested

tests/e2e/tests/meta_ha.rs runs three meta nodes, six OSDs and two gateways as separate processes, with writers running through both gateways:

  • the leader killed: writes resume within seconds, and every acknowledged write reads back through both gateways;
  • a follower killed and restarted: it catches up, then the leader is killed and the cluster carries on;
  • the leader frozen (SIGSTOP): a new leader is elected, and the old one steps down when it thaws;
  • both followers frozen: the leader, cut off, refuses reads sent to it directly (UNAVAILABLE), and serves again once they thaw;
  • two of three lost: writes refused, then resumed when one returns;
  • a KMS key, the objects encrypted under it, a multipart upload in progress, and the admin's credentials all survive a failover;
  • a meta node replaced by a new, empty one, which then forms part of the majority when another is lost.