Skip to content

Upgrades

Move a cluster to the next release one node at a time, under traffic. Design: core/upgrade-path.md.

What to know first

  • One release at a time. Go from each release to the next; to skip releases, upgrade through each in turn. Release candidates count as releases: 0.1.0-rc.1 goes to rc.2, rc.2 to rc.3, and the last candidate to 0.1.0. Finalize each upgrade before starting the next.
  • Format levels. Each release has a format level. During an upgrade, nodes on the new release read what it introduces but keep writing the previous release's formats, so you can still go back. The new formats start only when you finalize.
  • No going back after finalize. Back up meta first (stop a meta follower and copy its data directory, meta.redb and raft-log; an export command is planned).
  • Too-old nodes are refused. After finalize, meta refuses any node or client from an older release, and a node too old for the cluster stops at start-up with a message saying so, instead of running and misreading data.

The procedure

  1. Back up meta.
  2. Meta nodes: the followers one at a time, then the leader. After each, wait until it has rejoined: GET /status on its admin port shows a leader_id, and its own last_applied within a few entries of the leader's (GET /status on the leader). Each node's /status reports only its own indexes.
  3. OSDs, one failure domain at a time. After each, wait until it is registered and healthy (iolakectl node list). A restart needs no drain: reads tolerate up to m missing shards meanwhile.
  4. Gateways and block gateways, in any order.
  5. Check that every node runs the new release:
iolakectl upgrade status

It lists every node heard from in the last 90 seconds, with its release and format level, and says what blocks finalizing: a meta node or an OSD that hasn't reported (down, or still on a release from before format levels), or a node still on the old release. Gateways are stateless and can't be listed in advance, so check that every gateway you run appears. Each node reports every 30 seconds, and again within a few seconds when a report isn't taken (during an election); the meta leader answering counts itself as reported. A healthy cluster, a fresh one included, has no blockers, and can_finalize is false when there is nothing to raise.

  1. Finalize:
iolakectl upgrade finalize

It is refused while anything blocks it.

To go back before step 6, run the same steps in reverse order with the previous release's binaries.

How a node stops

Every service stops cleanly on SIGTERM, which Kubernetes, systemd and docker stop send, or on SIGINT (Ctrl-C), and exits 0:

  • Gateway: /_ready answers 503 at once, while the gateway goes on serving for --shutdown-delay-secs (default 5), so that load balancers and Kubernetes endpoints stop sending it requests first. Then it stops taking connections, gives the requests in flight up to --shutdown-drain-secs (default 20) to finish, and delivers what its audit and bucket-log spools hold, for up to --audit-drain-secs (default 20).
  • Meta: answers the calls in flight, stops Raft, and makes its state durable.
  • OSD: finishes the calls in flight. Every write it acknowledged is already on disk.
  • Block gateway: lets a flush under way finish (up to 10 s) and syncs its journal. Every write it acknowledged is already in the journal.

A second signal exits at once, without waiting for the rest (status 143 for SIGTERM, 130 for SIGINT). iolake-aio stops the same way, its gateway without the delay, and exits after 5 s at most.

Kubernetes

Roll the meta and OSD StatefulSets one pod at a time (partition), each gated on readiness, in the order above, then run iolakectl upgrade finalize.

A gateway pod's terminationGracePeriodSeconds must cover its shutdown: the delay, the request drain and the audit drain. The chart sets it to the audit drain plus 30 s when the audit spool is on; otherwise Kubernetes' default of 30 s covers the delay and the request drain. A pod still running when the grace period ends is killed, and an audit spool on an emptyDir goes with it.

API

Both calls are system admin only.

Call
GET /_admin/upgrade active_level, finalize_to, can_finalize, blockers, and nodes (kind, id, release, format level, address, seconds since last report)
POST /_admin/upgrade/finalize Raises the active level. Returns 409 with the blockers when not every node is ready

Clusters from before 0.1.0

0.1.0-rc.1 was the first release, at format level 8: no release runs in a cluster below it. Each release candidate upgrades to the next, as above, and the last one to 0.1.0.

A cluster on a development build from before 0.1.0-rc.1 (a format level below 8) can't be upgraded: a node of this release stops at start-up saying that the cluster "is at N: a development build from before 0.1.0, which can't be upgraded". Install the release fresh and copy the data across (with bucket replication, or any S3 tool).

A meta node of 0.1.0-rc.2 keeps its Raft log in its database until the cluster has its format level, which it gets when its first OSD registers, and moves it then. Upgrade a cluster once an OSD has registered: a meta node whose log is still in its database stops at start-up, saying so; run it on 0.1.0-rc.2 again until an OSD has registered, then upgrade it.

An rc.1 cluster on arm64, or built from source without the isal feature, is not upgraded in place. From 0.1.0-rc.2 every build erasure-codes with ISA-L; those builds coded MDS stripes with another code (reed-solomon-simd), whose parity later releases can't decode. A stripe that loses a data shard can't be rebuilt, and while old and new nodes mix, repair of a shard a write never stored rebuilds it in the other code, which reads then serve as data: the rolling-upgrade test, run that way, read objects back wrong. Install the latest release fresh and copy the data across, as above. A cluster on the amd64 rc.1 release binaries or image coded with ISA-L already, and upgrades in place.

An rc.1 cluster installed with the Helm chart's default protection (ec.k: 1, ec.m: 0) sets replication: 1 when it upgrades. From 0.1.0-rc.2, meta refuses to start with a default profile of no parity, and the chart's default is now 4+2 erasure coding over 6 OSDs, which such a cluster's default pool isn't; pass --set replication=1 (one copy) and its osd.replicas (Kubernetes). Its default pool, made 1+0, never took an object over the inline limit (4 KiB) and stays as it was made: install the latest release fresh and copy the objects across, as above.