Linux

Linux is the primary production target. Both amd64 and arm64 are first-class, and both erasure-code with ISA-L (AVX2, AVX-512 and GFNI on amd64; NEON and SVE on arm64). They write the same stripes, so amd64 and arm64 nodes mix in one cluster.

This page covers, in increasing order of commitment:

1. Release binary — iolake-aio, one process
2. A cluster under systemd — meta, OSDs and gateways as separate services
3. From source — Rust toolchain; also how you get the service binaries outside the container image

If you just want a quick local endpoint, use iolake-aio — one binary, no daemons. The Helm chart (Kubernetes) is the tested way to run separate services; this page covers doing it by hand.

---

System requirements

Component	Minimum	Recommended
Kernel	4.19+	5.15+ (for io_uring)
CPU	2 cores	8+ cores
RAM	4 GB	16+ GB (OSD), 8 GB (meta), 4 GB (gateway)
Disks (OSD)	1 × any block device	4+ dedicated raw disks per OSD node
Network	1 Gbps	10+ Gbps (especially between OSDs)

Filesystem: OSDs prefer raw block devices (/dev/sdb, /dev/nvme0n1). File-backed disks (/mnt/.../disk.raw) work but skip the O_DIRECT fast path and lose some durability guarantees. Disks are formatted with IOLake's own on-disk layout (superblock + WAL + blocks), not ext4/xfs — mkfs-ing first is not required and is not recommended for production OSDs.

Kernel features used:

• O_DIRECT on raw block devices (required for OSD disks)
• ioctl BLKGETSIZE64 for disk sizing
• io_uring when available (5.15+); falls back to threaded I/O, with a warning in the OSD's log, where the kernel or a seccomp profile denies it

---

1. Release binary

A release publishes iolake-aio only, for linux-amd64 and linux-arm64 (and darwin-arm64), named with the tag; there is no archive of the service binaries. Those are in the container image (ghcr.io/io-lake/iolake, see Docker) or built from source.

TAG=v0.1.0-rc.4
ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
BASE=https://github.com/io-lake/iolake/releases/download/$TAG
curl -LO $BASE/iolake-aio-$TAG-linux-$ARCH
curl -LO $BASE/iolake-aio-$TAG-linux-$ARCH.sha256
sha256sum -c iolake-aio-$TAG-linux-$ARCH.sha256
sudo install -m 0755 iolake-aio-$TAG-linux-$ARCH /usr/local/bin/iolake-aio
iolake-aio --version

releases/latest/download/… doesn't work for a pre-release such as v0.1.0-rc.4: name the tag.

The release binary needs glibc 2.28 or later and nothing else. It is built on AlmaLinux 8 (glibc 2.28); PCRE2 is linked in, and its TLS is rustls, so it loads no OpenSSL. The only shared libraries it loads are glibc's own, libgcc_s and the dynamic loader. By glibc, that covers (the kernel must meet the system requirements too):

Distribution	Release	glibc
RHEL, Rocky Linux, AlmaLinux	8 and later	2.28+
Debian	10 and later	2.28+
Ubuntu	20.04 and later	2.31+
Amazon Linux	2023	2.34
Fedora	29 and later	2.28+

Before a release is published, each Linux binary (amd64 and arm64) is checked to need no glibc symbol newer than 2.28 and no other library, and is run on Ubuntu 22.04, Debian 12, Rocky Linux 9 and Amazon Linux 2023. Amazon Linux 2 (glibc 2.26) and CentOS 7 (2.17) are too old: use the container image there.

Outbound HTTPS (to an OIDC provider, for one) trusts Mozilla's root certificates, which are built in, and the system's certificate store, so a private CA added there (update-ca-certificates, update-ca-trust) is trusted too.

v0.1.0-rc.1's binaries are the exception: built on Ubuntu 24.04, they need glibc 2.39, libssl3 and libpcre2-8-0.

---

2. A cluster under systemd

Every service is configured with flags; only iolake-osd also reads a TOML file (--config), which is where an OSD's failure domain (region, datacenter, rack, host) is set. The ports are the cluster's (ports), every binary's defaults.

iolake-install writes each host's units and OSD files and forms meta's Raft group: below. The same by hand follows it. Either way the binaries come from the image or a build, in /usr/local/bin.

With iolake-install

On each host, as root, with its own --node-id (1, 2, 3), the address the other hosts reach it at, every meta node's address, and its disks:

sudo iolake-install init --role all \
  --node-id 1 \
  --advertise node1.internal \
  --meta-endpoint http://node1.internal:9100,http://node2.internal:9100,http://node3.internal:9100 \
  --disks /dev/disk/by-id/nvme-A,/dev/disk/by-id/nvme-B \
  --ec-profile 4+2 \
  --systemd

It writes an OSD per disk (/etc/objectio/osd-0.toml, osd-1.toml, …, the units iolake-osd-0, iolake-osd-1, …), the units iolake-meta and iolake-gateway with their flags, and the data directories under /var/lib/objectio, and creates the system user objectio that meta and the gateway run as. It formats no disk: each must be blank, which its OSD formats when it first starts, or an IOLake disk. --ec-profile (or --replication <n>) sets the default pool (profiles allowed): it needs k+m OSDs, on as many hosts where there are that many. A host needn't run everything: --role meta,osd on the others, say, and the gateway on some (behind a load balancer). The flags include each port.

Start meta on every meta host, form the Raft group once, from any host, then start the rest:

sudo systemctl enable --now iolake-meta                  # every meta host
iolake-install cluster init node1.internal node2.internal node3.internal
sudo systemctl enable --now iolake-osd-0 iolake-osd-1 iolake-gateway

cluster init calls each meta node's admin port (9102), as the by-hand steps do, and is safe to run again. The units bind it on every interface (--admin-host 0.0.0.0), so it can run from any host. Without mTLS the port is plain HTTP with no authentication: keep it to the operators' network. With mTLS it is HTTPS, and changes membership only for a certificate of role admin; give cluster init one:

iolake-install cluster init --tls-cert admin.pem --tls-key admin.key --tls-ca ca.pem \
  node1.internal node2.internal node3.internal

The units carry no TLS flags of their own: under mTLS, give each service its certificate with a drop-in (systemctl edit iolake-meta, and so on) setting Environment=IOLAKE_TLS_CERT=… IOLAKE_TLS_KEY=… IOLAKE_TLS_CA=…, each its own role's (mTLS). The admin's keys are then in /var/lib/objectio/meta/admin-creds.env on every meta host.

One host is the same with one meta node: --role all, no --meta-endpoint (this host's meta), and an erasure code its disks can hold, one OSD each: three disks, --ec-profile 2+1; one disk, --replication 1. Then iolake-install cluster init 127.0.0.1.

For server-side encryption, put IOLAKE_MASTER_KEY=<base64 of 32 random bytes> in /etc/objectio/gateway.env (mode 0600), the same on every gateway, before starting it. Each OSD's failure domain goes in its file, [osd.failure_domain] (commented out as written); restart the OSD after changing it.

This path is tested end to end: one host, then three, set up by iolake-install, their units' commands run, the group formed by cluster init, then objects written and read through the gateway.

By hand

Service user and directories

sudo groupadd -r objectio
sudo useradd -r -g objectio -d /var/lib/objectio -s /usr/sbin/nologin objectio
sudo mkdir -p /etc/objectio /var/lib/objectio/meta /var/lib/objectio/osd
sudo chown -R objectio:objectio /var/lib/objectio

Meta (three nodes, Raft)

On each meta node, with its own --node-id (1, 2, 3) and the address the others reach it at:

iolake-meta \
  --node-id 1 \
  --listen 0.0.0.0:9100 \
  --raft-advertise meta1.internal:9100 \
  --metrics-port 9101 \
  --admin-port 9102 \
  --data-dir /var/lib/objectio/meta \
  --ec-k 4 --ec-m 2

--ec-k/--ec-m set the default pool's erasure code (profiles allowed); it needs at least k+m OSDs, on as many hosts. Once all three run, form the Raft group from node 1, over its admin port (plain HTTP, no authentication: keep 9102 to the operators' network; under mTLS HTTPS, with --cacert ca.pem --cert admin.pem --key admin.key). The admin port binds the host --listen names (--admin-host to change it): with --listen 0.0.0.0:9100, every interface.

curl -X POST http://meta1.internal:9102/init
curl -X POST http://meta1.internal:9102/add-learner \
  -H 'content-type: application/json' -d '{"node_id": 2, "addr": "meta2.internal:9100"}'
curl -X POST http://meta1.internal:9102/add-learner \
  -H 'content-type: application/json' -d '{"node_id": 3, "addr": "meta3.internal:9100"}'
curl -X POST http://meta1.internal:9102/change-membership \
  -H 'content-type: application/json' -d '{"voters": [1, 2, 3]}'
curl -s http://meta1.internal:9102/status    # leader, voters, learners

The first leader creates the admin user once, and every meta node writes its credentials to <data-dir>/admin-creds.env (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), readable by the service user only.

OSDs

One OSD per drive. /etc/objectio/osd.toml:

[osd]
listen = "0.0.0.0:9200"
advertise_addr = "http://osd1.internal:9200"
meta_endpoint = "http://meta1.internal:9100,http://meta2.internal:9100,http://meta3.internal:9100"

[osd.failure_domain]
datacenter = "dc1"
rack = "rack-a"
# host: the machine's hostname when left out

[storage]
disks = ["/dev/disk/by-id/nvme-…"]
data_dir = "/var/lib/objectio/osd"

iolake-osd --config /etc/objectio/osd.toml --metrics-port 9201

Flags given as well win over the file. A blank disk named in disks is formatted on first start; one that is neither blank nor this OSD's stops it (configuration).

Gateways

iolake-gateway \
  --listen 0.0.0.0:9000 \
  --meta-endpoint http://meta1.internal:9100,http://meta2.internal:9100,http://meta3.internal:9100 \
  --external-endpoint https://s3.example.com \
  --region us-east-1

Set IOLAKE_MASTER_KEY (base64 of 32 random bytes, the same on every gateway) for server-side encryption, and run several gateways behind a load balancer that checks /_ready. With more than one gateway, add --no-reexport-metrics and scrape meta and the OSDs directly (metrics).

---

3. Build from source

Needed if you want to track main, run a custom branch, or run the services outside the image. A binary needs the glibc it was built against or newer: to build one that runs where the release's does, build in an almalinux:8 container, with PCRE2_SYS_STATIC=1 to link PCRE2 in (the release workflow, .github/workflows/release.yml, is the recipe).

Toolchain

# Rust (1.93+)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"
rustup toolchain install 1.93

# Build deps
# Debian / Ubuntu
sudo apt-get install -y build-essential nasm autoconf automake libtool \
  libclang-dev pkg-config protobuf-compiler libprotobuf-dev cmake

# RHEL / Rocky / Alma 9+
sudo dnf install -y gcc-c++ make nasm autoconf automake libtool \
  clang-devel pkgconf protobuf-compiler protobuf-devel cmake

Every build erasure-codes with ISA-L, which is built from source with autotools (autoconf, automake, libtool) and bound with libclang. NASM is only required on x86_64, for ISA-L's x86 assembly; on ARM64 leave it out.

Console (SPA) prerequisites

iolake-aio and the gateway's /_console/ surface embed a pre-built React bundle. If you're only building server binaries and don't need the console, nothing to do. To build the console you need Node 22+:

# Ubuntu — NodeSource 22.x
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt-get install -y nodejs

cd console && npm ci && npm run build   # → console/dist

Build

git clone https://github.com/io-lake/iolake
cd iolake

cargo build --workspace --release

Binaries land in target/release/. For a system install:

sudo install -m 0755 target/release/{iolake-aio,iolake-gateway,iolake-meta,iolake-osd,iolakectl,iolake-install,iolake-block-gateway} \
  /usr/local/bin/

Then proceed as in section 2.

Run the test suite

cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check
# Or the one-shot:
make ci

---

Disk preparation — production checklist

• [ ] Dedicate whole disks to OSDs (no partitioning, no filesystem).
• [ ] Disable write-cache on SAS/SATA drives with non-battery-backed controllers: hdparm -W 0 /dev/sdX.
• [ ] For NVMe, confirm the drive supports FUA (Force Unit Access) — check nvme id-ctrl -H <dev> | grep -i fua.
• [ ] Raise /proc/sys/fs/file-max and ulimit -n (OSD opens one fd per concurrent operation × disk).
• [ ] Pin CPUs / set tuned-adm profile throughput-performance.
• [ ] If using io_uring, make sure the kernel is 5.15+ and /proc/sys/kernel/io_uring_disabled is 0, and that the OSD's log has no falling back to pread warning.

See topologies.md for how many disks and nodes to size for a given durability target.

---

systemd unit reference

The units iolake-install writes; to write them by hand, keep their shape. An OSD's, one per disk:

# /etc/systemd/system/iolake-osd-0.service
[Unit]
Description=IOLake OSD 0 (/dev/disk/by-id/nvme-A)
Documentation=https://iolake.dev/
After=network-online.target
Wants=network-online.target
After=iolake-meta.service

[Service]
Type=simple
# Opens raw block devices.
User=root
Group=root
ExecStart=/usr/local/bin/iolake-osd --config /etc/objectio/osd-0.toml --metrics-port 9201
Restart=on-failure
RestartSec=5s
# Every acknowledged write is already on disk; this is time to finish
# the calls in flight.
TimeoutStopSec=60
LimitNOFILE=1048576
LimitMEMLOCK=infinity
NoNewPrivileges=yes
ProtectHome=yes
PrivateTmp=yes
SyslogIdentifier=iolake-osd-0

[Install]
WantedBy=multi-user.target

Meta's and the gateway's run as objectio, with their flags on ExecStart and LimitNOFILE=65536, ProtectSystem=strict, meta's data directory its one writable path (ReadWritePaths=/var/lib/objectio/meta), and the gateway reading EnvironmentFile=-/etc/objectio/gateway.env. The After=iolake-meta.service line is there on a host that runs meta. Every service stops cleanly on SIGTERM (how a node stops); none reloads on SIGHUP, so there is no ExecReload.

---

Uninstall

sudo systemctl disable --now iolake-gateway iolake-meta iolake-osd-0 iolake-osd-1   # every OSD
sudo rm /etc/systemd/system/iolake-*.service
sudo systemctl daemon-reload
sudo rm /usr/local/bin/iolake-* /usr/local/bin/iolakectl
# Data + config (destructive):
sudo rm -rf /etc/objectio /var/lib/objectio /var/log/objectio
sudo userdel objectio && sudo groupdel objectio