Kubernetes¶

IOLake's Helm chart (deploy/helm/iolake/ in the source repo, published with each release as oci://ghcr.io/io-lake/charts/iolake) installs every component as one release: the gateway (Deployment), meta (StatefulSet with a Raft bootstrap Job), the OSDs (StatefulSet with per-pod volumes), and optionally the block gateway and a Prometheus + Grafana stack.

Its defaults are for production: 4+2 erasure coding over 6 OSDs, 3 metas, 2 gateways, authentication on. A laptop or CI cluster sets its own convenient values (the kind walkthrough does).

Audience	Start here
Trying it on a laptop	Kind walkthrough
Production install	Install the chart, then a production deployment
Tuning an existing release	Values reference

Prerequisites¶

Kubernetes 1.26+ (kubeVersion in Chart.yaml), Helm 3.
As many schedulable nodes as failure domains you want: the default 6 OSDs, one per node, survive two nodes lost at once.
A StorageClass with ReadWriteOnce volumes, ideally with WaitForFirstConsumer binding (so an OSD schedules on a node whose disk it can bind to); for raw block devices, one that supports volumeMode: Block. Local-path provisioners are for testing only.
For ingress: an ingress controller or a Gateway API implementation.
RBAC: the chart creates a ServiceAccount and Role for the gateway's host-lifecycle provider, and a ClusterRole reading nodes when OSDs take their failure domains from node labels.

No external dependencies (etcd, ZooKeeper, Redis): meta is a self-contained Raft cluster.

Installing the chart¶

helm install iolake oci://ghcr.io/io-lake/charts/iolake \
  --version 0.1.0 \
  --namespace iolake --create-namespace \
  --values my-values.yaml \
  --wait --timeout 15m

The chart's version pins the image to the same release. From a source checkout, use ./deploy/helm/iolake instead of the OCI reference.

--wait returns once every pod is ready (under a minute on a small cluster). The Raft bootstrap Job is created with the rest of the release, not as a post-install hook, so meta and the gateways become ready while Helm waits. A gateway that starts before meta is ready exits and is restarted, so on a first install the gateways show a restart or two.

Check it came up:

kubectl -n iolake get pods
# NAME                              READY   STATUS
# iolake-gateway-<hash>           1/1     Running
# iolake-meta-0                   1/1     Running
# iolake-meta-1                   1/1     Running
# iolake-meta-2                   1/1     Running
# iolake-meta-raft-bootstrap-1-…  0/1     Completed
# iolake-osd-0 … iolake-osd-5   1/1     Running

Workloads are named <release>-iolake-<component>, or <release>-<component> when the release name contains iolake (as above).

The first admin's access key is written by meta, on first boot, to /var/lib/objectio/meta/admin-creds.env in every meta pod (the secret never goes to the logs):

kubectl -n iolake exec iolake-meta-0 -- cat /var/lib/objectio/meta/admin-creds.env
# export AWS_ACCESS_KEY_ID=AKIA…
# export AWS_SECRET_ACCESS_KEY=…

Expose the gateway however your cluster does it (an ingress, type: LoadBalancer), or for a first look:

kubectl -n iolake port-forward svc/iolake-gateway 9000:9000 &
eval "$(kubectl -n iolake exec iolake-meta-0 -- cat /var/lib/objectio/meta/admin-creds.env)"
aws --endpoint-url http://localhost:9000 s3 ls

What the defaults protect¶

	Default
Erasure coding	`replication: 0`, `ec.k: 4`, `ec.m: 2`	The default pool survives any two OSDs lost at once
OSDs	`osd.replicas: 6`	The chart refuses to render with fewer than `ec.k + ec.m` (or than `replication` copies)
Meta	`meta.replicas: 3`	Survives one meta pod lost
Authentication	`gateway.noAuth: false`	Every request is signed (SigV4) or carries an OIDC/STS token
Disruption budgets	gateway `minAvailable: 1`; meta and OSD `maxUnavailable: 1`	A node drain takes one meta and one OSD at a time
Grafana (when enabled)	a random admin password, in a Secret	No well-known password

The chart refuses ec.m: 0: one unprotected copy of each object, for a one-OSD development cluster, is replication: 1, and its notes warn that one lost OSD then loses data. replication: 2 or 3 keeps that many copies instead of erasure coding.

A production deployment¶

deploy/helm/iolake/examples/production.yaml is a complete example: gateways behind an ingress, metas and OSDs spread over nodes and zones, OSDs on dedicated storage nodes with raw block volumes and their zone and rack read from node labels, an existing SSE key, NetworkPolicies, mTLS between the services, and the alert rules for a prometheus-operator. Copy it and change what it marks.

Where pods run¶

gateway, meta, osd and blockGateway each take nodeSelector, affinity, tolerations, topologySpreadConstraints, priorityClassName and podAnnotations, as in a pod spec. A spread constraint without a labelSelector gets the component's own pods' labels, so this is enough for one OSD per storage node:

osd:
  nodeSelector:
    iolake.io/storage: "true"
  tolerations:
    - { key: iolake.io/storage, operator: Equal, value: "true", effect: NoSchedule }
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule

global.imagePullSecrets goes to every pod.

Failure domains¶

Placement never puts two shards of a stripe in one failure domain at the level its pool spreads over (topologies). Each OSD registers where it sits:

host is always the Kubernetes node the pod runs on, so OSDs on one machine share a host.
region, zone, datacenter, rack are the fixed values under osd.failureDomain, the same for every OSD, unless nodeLabels names a node label to read a level from, per pod:

osd:
  failureDomain:
    region: eu-west-1
    datacenter: dc1
    nodeLabels:
      zone: topology.kubernetes.io/zone
      rack: topology.iolake.io/rack     # label your nodes with their rack

An init container reads the labels of the pod's node through the API server (the chart creates a ClusterRole allowing get on nodes, and only that container holds the token) and writes them into the OSD's configuration. An OSD on a node without a label it is told to read does not start: its node-topology init container fails, naming the node and the label, rather than place the OSD in a wrong failure domain.

OSD storage¶

Each OSD pod has two volumes: data, its disk, and state, a small volume for its metadata index. osd.persistence.data.volumeMode picks what the disk is:

`volumeMode`	The OSD's disk
`Filesystem` (default)	A file, `storage.diskFile`, on the volume; the OSD creates it on its first start
`Block`	The volume itself, a raw block device (local disks through a local-volume provisioner, a SAN), written directly with no filesystem in between

A blank device is formatted on the OSD's first start; a device that carries anything else (a filesystem, another cluster's disk) is refused, and the OSD doesn't start. With a filesystem volume, an OSD whose state volume has run before and whose disk file is missing doesn't start (a disk gone): to replace a data volume, see replacing a disk.

The data volumes' size is the cluster's capacity (hardware sizing).

Disk topology	StorageClass	Notes
Local NVMe through a local-volume provisioner	`WaitForFirstConsumer`, `volumeMode: Block`	One volume per disk; pin OSDs to the nodes that own them
Local NVMe through TopoLVM	`topolvm-provisioner`	LVM-backed, dynamic
Cloud block storage (EBS gp3, PD-SSD)	default	One volume, one OSD
Ceph RBD, NFS or object-backed volumes	don't	Storage on storage; the engine's durability assumes local block semantics

Probes¶

Every probe asks the service whether it can do its job, not only whether a port is open:

Component	Startup	Liveness	Readiness
Gateway		`GET /health` (port 9000)	`GET /_ready`: 503 until meta is reachable and enough OSDs are up for a write to meet quorum, and from the start of a shutdown
Meta	`GET /health` on the admin port (9102), up to 5 minutes to open its database	`GET /health`: 503 once its Raft has stopped (a fatal storage error, a panic), so the pod restarts	`GET /ready`: 503 until it is a member of the cluster with a leader known (before the bootstrap Job has added it, during an election)
OSD	`GET /health` on the metrics port (9201), up to 10 minutes to open its disk	the gRPC port (9200)	`GET /health`: 503 while a disk is unhealthy
Block gateway	the gRPC port (9300), up to 10 minutes to replay its journal	the gRPC port	the gRPC port: it opens only once the journal is replayed and the volumes are loaded

Under tls.enabled meta's admin port is HTTPS, and meta's probes ask over HTTPS (scheme: HTTPS). Kubelet presents no client certificate, and needs none: /health, /ready and /status answer any client, while the calls that change Raft membership take a certificate of role admin (mTLS).

A failed disk takes its OSD out of readiness without restarting it. Each probe is a value (meta.readinessProbe and so on) that can be replaced whole.

Meta and the OSDs start their pods in parallel (podManagementPolicy: Parallel): after a full restart a meta pod is ready only with a quorum, so pods started one after another, each waiting for the last to be ready, would never get past the first. Updates still roll one pod at a time, each waiting for readiness.

Disruption budgets¶

	Default
`gateway.podDisruptionBudget`	`minAvailable: 1`	Raise it with the replica count
`meta.podDisruptionBudget`	`maxUnavailable: 1`	Keeps a Raft majority
`osd.podDisruptionBudget`	`maxUnavailable: 1`	Keep it at or below `ec.m`

NetworkPolicy¶

On by default (networkPolicy.enabled): an ingress policy for meta, the OSDs, the gateway and the block gateway. Each component's pods take connections only on their own ports, and only from:

Ports	From
meta's gRPC (9100) and admin port (9102); the OSDs' gRPC (9200); the block gateway's gRPC (9300) and NBD (10809)	IOLake's own pods (meta, OSDs, gateways, block gateway) and the Raft bootstrap Job; `adminFrom` for meta's admin port, `blockGatewayFrom` for the block gateway's two
metrics: meta 9101, OSD 9201, block gateway 9301	every pod of the release (the chart's Prometheus included), and `metricsFrom`
the gateway's S3 port (9000, and `gateway.endpoints.external.port`)	every pod of the release; anyone, or with `gatewayFrom` set, those peers

Each ...From value is a list of NetworkPolicy peers:

networkPolicy:
  enabled: true
  gatewayFrom:            # the S3 port; empty: anyone
    - namespaceSelector:
        matchLabels: { kubernetes.io/metadata.name: ingress-nginx }
  blockGatewayFrom: []    # gRPC and NBD; empty: IOLake's pods only
  metricsFrom:            # /metrics of meta, the OSDs, the block gateway
    - namespaceSelector:
        matchLabels: { kubernetes.io/metadata.name: monitoring }
  adminFrom: []           # meta's admin port, e.g. a debugging pod reading /status

Egress isn't restricted (the OSDs' node-label lookup reaches the API server, the gateway an OIDC issuer or Vault). Kubelet's probes come from the pod's own node, which a NetworkPolicy doesn't filter.

The policies take effect only where the network plugin enforces NetworkPolicies (Calico, Cilium and most others; not every default plugin does). Without mTLS they are all that keeps the unauthenticated internal ports to IOLake: check that they are enforced, or turn mTLS on.

Secrets¶

Secret	Values
SSE master key	`gateway.sse.existingSecret` (key `existingSecretKey`, default `master-key`)	Or the chart makes `<release>-gateway-sse` once
OIDC client secret	`gateway.oidc.existingSecret` (key `client-secret`)	Or `gateway.oidc.clientSecret`
Delta Sharing secret key	`gateway.deltaSharing.existingSecret` (key `secret-key`)	Or `gateway.deltaSharing.secretKey`
Grafana admin password	`monitoring.grafana.existingSecret` (key `admin-password`)	Or the chart makes `<release>-grafana` with a random one
Vault token	`gateway.kms.vault.tokenSecretName` (key `token`)
mTLS certificates, one per role	`tls.secrets.<role>`, or made by cert-manager	mTLS

The SSE master key encrypts every object written to a bucket with default encryption; without it they can't be read. Create it yourself, once, and back it up:

kubectl -n iolake create secret generic iolake-sse \
  --from-literal=master-key="$(openssl rand -base64 32)"

gateway:
  sse:
    enabled: true
    existingSecret: iolake-sse

Left to the chart, the key is generated on the first install, read back (lookup) and reused on every helm upgrade, and kept on helm uninstall (helm.sh/resource-policy: keep), as is the Grafana Secret. helm template can't read the cluster, so a tool that deploys by rendering the chart (Argo CD, for one) would make a new key on every render: with those, use existingSecret.

Block gateway¶

blockGateway:
  enabled: true
  advertiseHost: block.example.com   # the host initiators attach to
  cacheBytes: 268435456              # 256 MiB write cache
  persistence:
    size: 5Gi
  service:
    type: LoadBalancer               # NBD is TCP: it must be reachable
    nbdPort: 10809
networkPolicy:
  blockGatewayFrom:                  # the machines that attach volumes
    - ipBlock: {cidr: 10.20.0.0/24}

It runs as a StatefulSet of one pod: its write journal holds every acknowledged write until it reaches the OSDs, and is used by one process at a time. A StatefulSet starts a replacement only once the old pod is gone, and an update stops the old pod first. The journal's volume is kept on helm uninstall.

NBD has no authentication: whoever reaches port 10809 and knows a volume's name can attach it (listing the names is refused). Keep it to the machines that attach volumes, with networkPolicy.blockGatewayFrom and, for a LoadBalancer, the load balancer's source ranges. Under mTLS the block gateway's gRPC (9300) takes only a certificate of role admin (iolakectl's block commands).

Monitoring¶

The chart's own Prometheus and Grafana (monitoring.enabled: true) are for a quick look. In a cluster with a prometheus-operator, turn on the alert rules as a PrometheusRule (monitoring.prometheusRule.enabled, with labels matching its rule selector) and scrape meta's :9101, the OSDs' :9201 and the gateways' :9000/metrics; the chart has no ServiceMonitor. See monitoring.

Values reference¶

The full defaults, with comments, are in deploy/helm/iolake/values.yaml. The keys an install usually sets:

Key	Default
`replication`	`0`	The default pool's protection: 0 erasure-codes it `ec.k`+`ec.m`; n keeps n copies of each object
`ec.k`, `ec.m`	`4`, `2`	The erasure code (`ec.m` at least 1); the block gateway codes volumes with it whatever `replication` is
`osd.replicas`	`6`	At least `ec.k + ec.m` (or `replication`); more lets a stripe be rebuilt elsewhere while an OSD is down
`osd.persistence.data.size`, `.storageClass`, `.volumeMode`	`50Gi`, the cluster's default, `Filesystem`	Each OSD's disk
`osd.persistence.state.size`	`5Gi`	Each OSD's metadata index
`osd.failureDomain.{region,zone,datacenter,rack}`	`default`, empty, `default`, `rack1`	Fixed levels, the same for every OSD
`osd.failureDomain.nodeLabels`	`{}`	Levels read from each OSD's node labels
`osd.podDisruptionBudget.maxUnavailable`	`1`
`meta.replicas`	`3`	1 (no fault tolerance), 3 or 5; not an even number
`meta.persistence.size`	`10Gi`
`gateway.replicas`	`2`	Or `gateway.autoscaling` (an HPA on CPU)
`gateway.noAuth`	`false`	`true` serves every request unauthenticated, as the admin: development only
`gateway.externalUrl`	empty	The public URL (Iceberg vended credentials, Delta Sharing, the console's OIDC callback)
`gateway.s3Domains`	`[]`	Domains buckets are addressed under by host name (`--s3-domain`); each needs wildcard DNS, an ingress host `*.<domain>` and a certificate covering it (virtual-hosted addressing)
`gateway.ingress`	off	`className`, `hosts`, `tls`, `annotations`
`gateway.oidc`	off	`issuerUrl`, `clientId`, `clientSecret` or `existingSecret`, `adminRoles`
`gateway.sse`	off	`existingSecret`, or a key the chart makes
`gateway.trustedProxies`	empty	Proxies whose `X-Forwarded-For` is believed (`aws:SourceIp`)
`gateway.endpoints`	unnamed	Named endpoints for `aws:SourceVpce`; a separate external port for the ingress
`gateway.topology`	empty	The gateway's own position, for locality-aware reads
`gateway.audit`	spool on, bucket `objectio-audit`, 365 days	Audit
`gateway.hostProvider`	`k8s`	Lets the admin API add OSD pods and restart them
`<component>.nodeSelector`, `.affinity`, `.tolerations`, `.topologySpreadConstraints`, `.priorityClassName`, `.podAnnotations`	empty	Where pods run
`<component>.startupProbe`, `.livenessProbe`, `.readinessProbe`	see Probes
`<component>.resources`	per component
`networkPolicy.*`	on	NetworkPolicy
`tls.enabled`	`false`	mTLS between the services
`blockGateway.enabled`	`false`	Block gateway
`monitoring.enabled`, `monitoring.prometheusRule.enabled`	`false`	Monitoring
`global.imagePullSecrets`	`[]`
`global.toolsImage`	`curlimages/curl:8.10.1`	sh and curl, for the bootstrap Job and the node-label lookup; point it at a mirror when air-gapped

Minimum OSDs for common erasure codes:

Profile	OSDs	Recommended
4+2	6	7–8
8+4	12	13–14
LRC 10+2+2	14	15

The profiles a pool may have are in data protection.

Scaling an existing release¶

Add OSDs by raising osd.replicas:

helm upgrade iolake oci://ghcr.io/io-lake/charts/iolake --version 0.1.0 \
  -n iolake --reuse-values --set osd.replicas=9

New OSDs register with meta and take placement groups; see failure recovery for how data moves.

Meta membership: every install and upgrade runs the Raft bootstrap Job (<release>-meta-raft-bootstrap-<revision>). On the first install it makes the cluster; on an upgrade it adds, through the leader, the pods that aren't voters yet (1 → 3, 3 → 5); with every pod a voter it changes nothing. It doesn't scale meta down: call /change-membership with the smaller voter set first, then lower meta.replicas (meta HA). With tls.enabled the Job calls meta's admin port over HTTPS with the admin role's certificate, the only one that port takes membership changes from.

Losing a meta pod¶

With three metas, any one pod can go (killed, evicted, rescheduled, or rolled by kubectl rollout restart) without losing anything the gateway acknowledged. If it was the leader, the other two elect one in about 5 seconds; writes that arrive meanwhile get 503 ServiceUnavailable, which S3 clients retry, and resume on their own. The pod comes back on its volume, catches up, and is ready again once it knows the leader. Losing two of three at once stops writes until one returns: keep the PodDisruptionBudget (meta.podDisruptionBudget) on.

Upgrading from 0.1.0-rc.3¶

0.1.0-rc.3 was published as ObjectIO: its chart was oci://ghcr.io/object-io/charts/objectio and its image ghcr.io/object-io/objectio. The chart is now iolake, at oci://ghcr.io/io-lake/charts/iolake, and its image ghcr.io/io-lake/iolake.

The chart names every resource after the release and the chart's name (<release>-<chart>, or <release> when the release name contains the chart's). Upgraded to the renamed chart as it is, a release would get new names: new StatefulSets whose pods start on new, empty volumes, and a newly generated SSE master key, beside the old ones. Keep the old names by setting nameOverride to the old chart's name (here for a release installed as the guide said, objectio in the namespace objectio; a release that already sets nameOverride or fullnameOverride keeps its own and needs nothing more):

helm upgrade objectio oci://ghcr.io/io-lake/charts/iolake --version <version> \
  -n objectio --reuse-values --set nameOverride=objectio

Every name stays as it was: the StatefulSets and their volumes, the pods' labels (a StatefulSet's selector can't change), the Services, and the Secrets the chart made (the SSE key, Grafana's password, the certificates). The pods roll onto the new image as in any upgrade (upgrades). Keep nameOverride: objectio in the release's values from then on.

Before the upgrade, change in the release's values whatever names the old product:

The image, if your values name it: global.imageRegistry to ghcr.io/io-lake and global.image.repository to iolake. Left to the chart, it is already the new one.
Any environment variable you give the services: OBJECTIO_* is now IOLAKE_*, and a service ignores the old name.
Nothing for node labels: the chart reads the labels your values name, so objectio.io/storage and topology.objectio.io/rack keep working. The example values now use iolake.io/storage and topology.iolake.io/rack.

The Prometheus and Grafana the chart can run take the new metric names and alert rules with the upgrade; dashboards and alerts of your own need the new names (upgrading a 0.1.0-rc.3 cluster).

Upgrading from 0.1.0-rc.1¶

The names below are those of a release installed as the guide then said, before the rename: objectio, in the namespace objectio.

The 0.1.0 chart changes five things an upgrade has to handle:

Meta and the OSDs start their pods in parallel, a StatefulSet field an upgrade can't change. Delete the two StatefulSets, leaving their pods running, before the upgrade; it recreates them and rolls the pods one at a time:

kubectl -n objectio delete statefulset objectio-meta objectio-osd --cascade=orphan

The block gateway is a StatefulSet, no longer a Deployment. Stop it before the upgrade, so that two pods never use its journal at once: kubectl -n objectio scale deployment objectio-block-gateway --replicas=0.
The defaults changed (4+2 over 6 OSDs, authentication on). A release installed with the old defaults (1+0, one OSD, no auth) sets replication: 1, its osd.replicas and, until its clients sign their requests, gateway.noAuth: true when it upgrades (upgrades).
NetworkPolicies are on by default: the internal ports take only IOLake's own pods. Clients that attach block volumes from outside the release are listed in networkPolicy.blockGatewayFrom before the upgrade, or they lose their NBD connections (NetworkPolicy).
With tls.enabled, the chart issues a certificate per role and rolls each component onto its own. Finalize the upgrade only once every pod runs on its role's certificate: after obioctl upgrade finalize, a certificate that names no role is refused everything (mTLS).

Uninstalling¶

helm uninstall iolake -n iolake

Helm leaves the meta and OSD volumes (StatefulSet claims), the block gateway's journal volume, and the SSE and Grafana Secrets the chart made. Reinstalling under the same release name and namespace finds them and uses them again. To remove everything:

kubectl -n iolake delete pvc -l app.kubernetes.io/instance=iolake
kubectl -n iolake delete secret iolake-gateway-sse iolake-grafana --ignore-not-found
kubectl delete namespace iolake

Kind (local Kubernetes) walkthrough¶

deploy/kind/setup.sh stands up a 3-node Kind cluster (1 control-plane + 2 workers), builds every image, loads them into the cluster, installs the chart with deploy/kind/values.yaml, port-forwards the gateway to localhost:9000, and runs a smoke test covering S3, Iceberg and Delta Sharing.

# Prereqs: docker, kind, kubectl, helm
git clone https://github.com/io-lake/iolake
cd iolake

make kind-up                                 # build + deploy from source
# OR
make kind-up-registry                        # pull from ghcr.io/io-lake (no local build)

If something already listens on port 9000, forward the gateway elsewhere: PF_PORT=19000 make kind-up.

To check meta HA on it, with S3 traffic running from a pod: kill the meta leader's pod, then a follower's, then roll the StatefulSet; every acknowledged write must read back.

deploy/kind/meta-ha-test.sh

When it finishes, the script prints:

Endpoints (via kubectl port-forward):
  S3 / Iceberg / Delta Sharing  http://localhost:9000
  Grafana                       http://localhost:3000   (admin / iolake)

EC scheme: 4+2  |  Meta: 3 nodes  |  OSDs: 6 nodes

Teardown and re-deploy¶

make kind-down             # delete the whole Kind cluster
make kind-load             # rebuild images + reload into a running cluster
kubectl -n iolake rollout restart statefulset/iolake-osd  # pick up new images

What the kind values change¶

deploy/kind/values.yaml sets, for a throwaway cluster:

gateway.noAuth: true: every request is served as the admin, unsigned (the smoke test uses --no-sign-request). Never use these values on a real cluster.
Grafana's password iolake, one gateway, and small volumes on Kind's standard (local-path) StorageClass, which is not durable.
4+2 over 6 OSDs on 2 workers: several OSDs share a node, so the cluster doesn't show what losing a node does.

For anything else, start from the chart's defaults or examples/production.yaml.

Troubleshooting¶

OSD pods stuck `Pending`¶

Usually a volume that can't bind: kubectl -n iolake describe pvc. The common causes: no default StorageClass, a class without volumeMode: Block support, or WaitForFirstConsumer with a nodeSelector or spread constraint that no node with a free disk satisfies.

An OSD stuck in `Init:Error` or `Init:CrashLoopBackOff`¶

Its node-topology init container couldn't read a failure-domain level:

kubectl -n iolake logs iolake-osd-3 -c node-topology
# node worker-7 has no label topology.iolake.io/rack (osd.failureDomain.nodeLabels.rack): …

Label the node (kubectl label node worker-7 topology.iolake.io/rack=r12) and delete the pod.

Meta pods `Running` but not `Ready`¶

A meta pod is ready once it is a member of the cluster with a leader known. Its /ready says what is missing:

kubectl -n iolake port-forward pod/iolake-meta-0 9102:9102 &
curl -s localhost:9102/ready; echo
curl -s localhost:9102/status

With tls.enabled it is HTTPS, and reading needs no client certificate, only the CA (the meta certificate names localhost):

kubectl -n iolake get secret iolake-tls-meta -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
curl -s --cacert ca.crt https://localhost:9102/ready; echo

"not a member of a raft cluster" means the bootstrap Job hasn't added it. Check the Job:

kubectl -n iolake get jobs -l app.kubernetes.io/component=meta-raft-bootstrap
kubectl -n iolake logs job/iolake-meta-raft-bootstrap-<revision>

The Job waits for every meta pod's admin port; a NetworkPolicy of your own that blocks pod-to-pod traffic in the namespace stops it. A failed Job runs again with the next helm upgrade.

An OSD `Running` but not `Ready`¶

Its /health (metrics port) reports a disk that isn't healthy; the OSD's log says which and why. See failure recovery.

`kubectl port-forward` drops connections¶

Use a Service of type: LoadBalancer (on clouds) or an ingress in production. port-forward is for development only.