Mutual TLS between the services¶

With mTLS on, every gRPC hop between IOLake's components runs over mutual TLS, and each side proves it holds a certificate signed by the cluster's CA:

gateway to meta and to the OSDs;
OSD to meta;
meta to meta, including Raft;
meta to the OSDs (liveness, repair, drain);
block gateway to meta and to the OSDs;
iolakectl to the block gateway (volume, snapshot).

A peer without such a certificate is refused when it connects: a plaintext client, a TLS client with no certificate, or one with a certificate from another CA. The node that refused it logs a warning naming the peer's address (gRPC: refused <address>: TLS handshake: …). A peer that connects and never completes the handshake is disconnected 10 s after it connected, on every gRPC port and on meta's admin port, so it holds neither a connection nor anything else on the node. The other way round, a node connecting to another gives up when its connect timeout is out (5 s, less for the probes), the TLS handshake included: a peer that takes the connection and then answers nothing, such as a frozen process, fails the attempt rather than holding it.

Each certificate also names its role, and each call is checked against it: an OSD's certificate can't read a user's secret key from meta, a gateway's can't purge an OSD. Meta's admin port (Raft bootstrap and membership) is HTTPS with the same certificates, and changes membership only for the admin role.

TLS for clients (S3, the console, Iceberg) is a separate matter: it's terminated at the ingress or load balancer.

Without mTLS¶

Without mTLS nothing between the services is authenticated. Anyone who can reach meta's gRPC port (9100) can read every user's secret key and the key STS and console sessions are signed with; anyone who reaches an OSD (9200) can write, overwrite and delete shards; anyone who reaches meta's admin port (9102) can add a Raft member and be sent all the metadata. Each node says so in its log when it starts.

That is fine for development and for the all-in-one on a laptop, which keeps its internal ports on loopback. Anywhere else, turn mTLS on, or run on a network that only IOLake's own nodes can reach (on Kubernetes the chart's network policies, on by default; elsewhere a firewall) and keep it that way. The ports are in the installation guide.

Roles¶

A certificate names its role in a SAN URI, spiffe://objectio/<role>:

Role	Presented by	May call
`meta`	meta nodes	everything on meta (a follower forwards each call to the leader under its own certificate, after checking the caller's); Raft; everything on the OSDs: repair, recovery, drain, placement groups, scrub, purge
`osd`	OSDs	on meta: registering, its placement groups, reporting its version, the liveness probe
`gateway`	S3 gateways	on meta: everything but registering OSDs and the block gateway's volumes, users' secret keys and the STS signing key included; on the OSDs: reading and writing shards and object metadata, deleting shards, listings, status
`block-gateway`	block gateways	on meta: its volumes and chunk maps, where to place them, reporting its version; on the OSDs: writing, reading and deleting shards
`admin`	operators: `iolakectl`'s block commands, the Helm Raft bootstrap Job, `iolake-install cluster init`	the block gateway's gRPC (volumes, snapshots); changing Raft membership on meta's admin port

A call no role of the caller's is allowed, or one the server doesn't know, is refused with PERMISSION_DENIED. A process that is several components presents one certificate naming several roles: the all-in-one's names all five. A certificate naming no role is refused everything (but see upgrading).

Turning it on¶

Every component takes the same three settings, as flags or environment variables:

Flag	Environment	What
`--tls-cert`	`IOLAKE_TLS_CERT`	This node's certificate (PEM), naming its role. It is presented both as a server and as a client.
`--tls-key`	`IOLAKE_TLS_KEY`	Its private key (PEM)
`--tls-ca`	`IOLAKE_TLS_CA`	The CA (PEM) that signs every node's certificate. It is the only CA trusted.

The three go together. iolake-aio takes them too: its own internal hops then use mTLS, and meta's admin port inside it is HTTPS.

A server's certificate must name every address peers dial it by: the addresses registered as --advertise-addr (OSDs), --raft-advertise (meta) and the --meta-endpoint targets. Names and IPs both work. Give each role its own certificate, naming only that role's addresses, so one role's certificate can't stand in for another's, as a client or as a server.

Enable it at install time. A cluster can't run half on TLS and half off, so turning it on, or off, later means restarting every component at once.

Issuing the certificates yourself¶

With a CA of your own (ca.pem, ca.key), one certificate per role; the OSDs', for example:

openssl req -new -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout osd.key -subj "/CN=iolake-osd" -out osd.csr
openssl x509 -req -in osd.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
  -days 365 -out osd.pem -extfile <(printf '%s\n' \
  "subjectAltName=URI:spiffe://objectio/osd,DNS:osd1.example.internal,DNS:osd2.example.internal,IP:127.0.0.1" \
  "extendedKeyUsage=serverAuth,clientAuth")

The admin certificate needs only its URI: it is only ever a client's.

Meta's admin port¶

Under mTLS meta's admin port serves HTTPS with the meta node's certificate. A client certificate is optional when connecting, and checked for each call:

Call	Takes
`GET /status`, `/health`, `/ready`	any client, with a certificate or none: they only read the node's state, and kubelet's probes present no certificate
`POST /init`, `/add-learner`, `/change-membership`, and any other call	only a client certificate of role `admin`; anything else is answered 403

curl --cacert ca.pem --cert admin.pem --key admin.key \
  -X POST https://meta1:9102/add-learner -H 'content-type: application/json' \
  -d '{"node_id": 2, "addr": "meta2:9100"}'

/status stays open because it holds nothing a peer can't learn otherwise (node ids, their Raft addresses, the term and log index), and operators and their scripts read it without an admin certificate.

The port binds the host meta's gRPC listens on (--listen), or --admin-host. iolake-install cluster init forms the group over it with --tls-cert, --tls-key and --tls-ca, an admin certificate (Linux); the chart's bootstrap Job with the admin Secret, and meta's probes ask over HTTPS (scheme: HTTPS).

Kubernetes¶

The chart requests a certificate per role from cert-manager, each with its role's URI and the names of that role's pods and services, mounts each component's own, and sets the variables above. The Raft bootstrap Job gets the admin one.

tls:
  enabled: true
  # Either name a cert-manager CA issuer, and the chart requests the
  # certificates (Secrets <release>-iolake-tls-<role>):
  certManager:
    issuerRef: {name: iolake-ca, kind: Issuer}
  # or create the Secrets yourself (tls.crt, tls.key and ca.crt each),
  # one per role, and name them:
  # secrets:
  #   meta: my-meta-tls
  #   osd: my-osd-tls
  #   gateway: my-gateway-tls
  #   block-gateway: my-block-tls
  #   admin: my-admin-tls

A CA issuer is what you want here: ca.crt is then the CA that signs each certificate. A minimal one, with cert-manager installed:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata: {name: iolake-selfsigned}
spec: {selfSigned: {}}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: {name: iolake-ca}
spec:
  isCA: true
  commonName: iolake-ca
  secretName: iolake-ca
  issuerRef: {name: iolake-selfsigned, kind: Issuer}
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata: {name: iolake-ca}
spec: {ca: {secretName: iolake-ca}}

Upgrading from a release whose certificates name no role¶

A release before roles used one certificate, naming none, for every component. During the rolling upgrade, while the cluster's format level is still the previous release's, a certificate naming no role is taken for any role, so the nodes not yet upgraded keep working. Once the upgrade is finalized (iolakectl upgrade finalize), such a certificate is refused everything. The chart's upgrade issues the per-role certificates and rolls each component onto its own; with certificates of your own, reissue them with roles before finalizing. Then delete the old shared certificate and its Secret.

Renewal¶

Components read their certificate when they start. cert-manager renews the Secrets 30 days before the certificates expire, and the chart's default lifetime is one year. After a renewal, restart the components one at a time so they pick up the new certificates:

kubectl rollout restart statefulset/<release>-iolake-meta statefulset/<release>-iolake-osd \
  deployment/<release>-iolake-gateway

Picking up a renewed certificate without a restart is planned.

How it's tested¶

The end-to-end suite runs with mTLS on: every node of every test cluster gets a test CA's certificate for its role. Tests check that:

a node refuses a plaintext client, a client with no certificate and one with another CA's certificate, each within a bound, and serves the cluster's own;
meta and an OSD disconnect a client that connects and never starts a TLS handshake once the handshake timeout is out;
meta refuses the STS signing key and a user's secret key to every role but a gateway's, and to a certificate naming none, and an OSD refuses placement-group calls and purges from anyone but meta;
meta's admin port refuses plain HTTP, and /init and membership changes from a client with no certificate or another role's, while /status, /health and /ready answer a client with no certificate;
iolake-install cluster init forms nothing without a certificate or with another role's, and forms the group with an admin one;
the all-in-one, serving S3 on every interface, keeps every other port on loopback.

Unit tests check that every method of every service has a rule, that a certificate naming no role is any role only before the upgrade is finalized, and that on the admin port only reading the node's state is open to a client without an admin certificate. Others check that a gRPC server serves only a peer with a certificate the CA signed and hands that certificate to the call, that a peer silent or stopped part way through its handshake is disconnected while others are served, and that connecting to a server that never answers the handshake fails at the connect timeout. The rolling-upgrade test runs plain. IOLAKE_E2E_PLAIN=1 runs the whole suite plain; the mTLS tests then fail, since they have nothing to test.