Skip to content

Docker

IOLake ships one multi-stage Dockerfile. Its all target is the image that is published and that the Helm chart runs; per-service targets can be built locally.

There is no Compose file for running a cluster: the repository's docker-compose.yml holds development tools (build, test, lint), and deploy/monitoring/ a Prometheus and Grafana.

Image targets

Target Binary Ports Role
gateway iolake-gateway 9000 S3 / Iceberg / Delta Sharing / /_console/
meta iolake-meta 9100 (gRPC and Raft), 9101 (metrics), 9102 (admin) Raft metadata
osd iolake-osd 9200 (gRPC), 9201 (metrics) Object storage daemon
cli iolakectl — Admin CLI
block-gateway iolake-block-gateway 9300 (gRPC), 9301 (metrics), 10809 (NBD) Block volumes over NBD and gRPC
all gateway, meta, osd, iolakectl, iolake-install, block-gateway, and the console 9000 + internal One image, any service

Port numbers are the cluster's, used by the Helm chart and every binary's defaults (ports).

One multi-arch image (linux/amd64, linux/arm64) is published on GHCR, and it carries every service:

  • ghcr.io/io-lake/iolake:<tag> (for example v0.1.0-rc.4), also tagged latest — the all target

It has no entrypoint: name the binary to run, as the Helm chart does. The per-service targets (gateway, meta, osd, cli, block-gateway) are not published; build them locally if you want them. Each runs its binary with the cluster's ports by default and checks its own health (health checks):

docker build --target gateway -t iolake-gateway .

Both linux/amd64 and linux/arm64 erasure-code with ISA-L and write the same stripes, so a cluster may mix them.

The published image

The all target packages the service binaries plus the console SPA. You choose the binary at docker run time:

# The admin CLI against an existing cluster
docker run --rm --network host \
  -e IOLAKE_ACCESS_KEY -e IOLAKE_SECRET_KEY \
  ghcr.io/io-lake/iolake:v0.1.0-rc.4 \
  iolakectl --endpoint http://localhost:9000 user list

# A service: the flags are the binary's own (see configuration.md)
docker run --rm ghcr.io/io-lake/iolake:v0.1.0-rc.4 iolake-gateway --help

The image runs as the non-root user objectio, declares /var/lib/objectio as a volume, and has no curl.

Running the services in containers

Without Kubernetes, run each service from the image with the flags in Linux: meta with --node-id, --raft-advertise and its data directory on a volume, then the Raft bootstrap calls to its admin port; OSDs with an osd.toml mounted at /etc/objectio/osd.toml; gateways with every meta node's address in --meta-endpoint.

An OSD on raw block devices needs them passed as devices: and the capability to open them with O_DIRECT:

services:
  osd1:
    image: ghcr.io/io-lake/iolake:v0.1.0-rc.4
    command: ["iolake-osd", "--config", "/etc/objectio/osd.toml"]
    cap_add: [SYS_RAWIO]
    devices: [/dev/nvme1n1:/dev/nvme1n1]
    volumes:
      - ./osd1.toml:/etc/objectio/osd.toml:ro
      - osd1-state:/var/lib/objectio/osd

Building images locally

# Single-arch, native platform
make docker                 # gateway, meta, osd, cli and all
make docker-gateway         # just the gateway
make docker-all             # the published image's target

# Multi-arch (requires docker buildx, pushes to registry)
DOCKER_REGISTRY=ghcr.io/myorg/ DOCKER_TAG=v1.0.0 \
  make docker-multiarch

# Multi-arch local-only (current platform, loads into Docker)
make docker-multiarch-local

Build args exposed by the Dockerfile:

Arg Default Notes
RUST_VERSION 1.94 Rust compiler in the builder-base stage (matches rust-toolchain.toml)
FEATURES io-uring Cargo features to build with. ISA-L erasure coding is not one: every build has it.
TARGETPLATFORM / TARGETARCH from buildx Set automatically by docker buildx

Environment variables (containerised services)

Most configuration goes via CLI flags in the container's command. A set of settings is also reachable via env vars (the full list is in configuration.md):

Env var Service Purpose
IOLAKE_TOPOLOGY_REGION gateway Sets --topology-region
IOLAKE_TOPOLOGY_ZONE gateway Sets --topology-zone
IOLAKE_TOPOLOGY_DATACENTER gateway Sets --topology-datacenter
IOLAKE_TOPOLOGY_RACK gateway Sets --topology-rack
IOLAKE_TOPOLOGY_HOST gateway Sets --topology-host
IOLAKE_HOST_PROVIDER gateway Sets --host-provider (noop | k8s)
IOLAKE_HOST_PROVIDER_OSD_STS gateway OSD StatefulSet name (k8s provider only)
POD_NAMESPACE gateway Namespace for host-provider RBAC (k8s only)
IOLAKE_MASTER_KEY gateway SSE-S3 master key (base64 32-byte)
IOLAKE_CONSOLE_DIR gateway Override the /_console/ static-asset dir
IOLAKE_TLS_CERT, IOLAKE_TLS_KEY, IOLAKE_TLS_CA all mTLS between services
RUST_LOG all Overrides --log-level; same syntax as tracing

Health checks

Each service image checks its own /health (see health endpoints) with healthcheck.sh PORT, a GET over bash's /dev/tcp (the images carry no HTTP client):

Image Checks Default command
gateway :9000/health (/_ready is the readiness check) --listen 0.0.0.0:9000
meta :9101/health --listen 0.0.0.0:9100 --data-dir /var/lib/objectio/meta
osd :9201/health (503 once a disk has had an I/O or checksum error) --listen 0.0.0.0:9200 --data-dir /var/lib/objectio/osd
block-gateway :9301/health --listen 0.0.0.0:9300 --nbd-listen 0.0.0.0:10809 --data-dir /var/lib/objectio/block-gw

A command that moves a port (--listen, --metrics-port, --metrics-listen) moves what the check must ask: override the HEALTHCHECK with it. The all image (the published one) declares no HEALTHCHECK: it runs any service, so use these endpoints in your orchestrator; the Helm chart has its own probes.

Troubleshooting

"operation not permitted" on an OSD

OSDs need CAP_SYS_RAWIO to open raw block devices with O_DIRECT (cap_add: [SYS_RAWIO], or privileged: true).

"falling back to pread" in an OSD's log

The image does its disk I/O through io_uring, and docker's default seccomp profile denies the io_uring system calls. The OSD (or aio) then logs storage error: io_uring setup for <device>: Operation not permitted (os error 1); falling back to pread for each device and serves with pread: correct, but slower. A privileged: true container, or a seccomp profile that allows io_uring_setup, io_uring_enter and io_uring_register (--security-opt seccomp=<file>), gets io_uring.

Gateway can't reach meta

Gateways and OSDs take every meta node's address in one comma-separated --meta-endpoint (for example http://meta1:9100,http://meta2:9100,http://meta3:9100); any one that answers serves them. Check that the names resolve inside the containers' network and that each meta node's --raft-advertise is an address the others can reach, not 0.0.0.0.