IOLake Deployment Options¶
This guide covers deployment configurations based on your hardware topology, in detail; topologies.md is the overview.
Quick Reference¶
| Nodes | Disks/Node | Recommended Config | Efficiency | Survives |
|---|---|---|---|---|
| 1 | 12 | EC 4+2 (disk-level) | 66% | 2 disk failures |
| 3 | 12 | EC 6+3 (node-level) | 66% | 1 node failure |
| 3 | 12 | Replication 3-way | 33% | 2 node failures |
| 6+ | 6+ | LRC 6+2+2 (rack-level) | 60% | 1 rack + 1 disk |
| 10+ | 10+ | LRC 10+2+2 (rack-level) | 71% | 1 rack + 1 disk |
Failure Domain Hierarchy¶
Region (geo-redundancy)
└── Datacenter (power/network)
└── Rack (ToR switch, PDU)
└── Node (server)
└── Disk (SSD/HDD)
CRUSH spreads shards across the highest available failure domain to maximize fault tolerance.
Topology Configuration¶
Each OSD reports its failure domain on registration. It is set only in
the OSD's /etc/objectio/osd.toml (no flag sets it):
[osd]
listen = "0.0.0.0:9200"
advertise_addr = "http://osd1.internal:9200"
meta_endpoint = "http://meta1.internal:9100,http://meta2.internal:9100,http://meta3.internal:9100"
# Failure domain - where this OSD sits in the hierarchy
[osd.failure_domain]
region = "us-east" # Geographic region
datacenter = "dc1" # Data center or AZ
rack = "rack-01" # Physical rack
# host defaults to the machine's hostname
[storage]
disks = ["/dev/disk/by-id/nvme-…"] # one drive per OSD
data_dir = "/var/lib/objectio/osd"
How placement uses this:
- OSDs register with their failure domain.
- A pool names a failure-domain level:
host(the default),node(each OSD),rack,datacenter,zoneorregion. - Each placement group's shards go to distinct domains at that level:
one per domain, or, with
--spread-domains N --per-domain P, over N domains with at most P in each. - The default pool is made at
hostlevel when there are at least k+m hosts, else atnodelevel.
The "Configuration" commands below are iolakectl pool create calls
(CLI); a bucket goes to a pool with
iolakectl bucket create <name> --pool <pool> (or the x-iolake-pool
header on CreateBucket), fixed for its life.
Scenario 1: Single Node (12 Disks)¶
Hardware: 1 server with 12 disks
Failure Domain: Disk-level only (can't survive node failure)
Option 1A: EC 4+2 (Recommended)¶
┌─────────────────────────────────────────────────────────────────────────┐
│ Single Node │
│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │
│ │ D0 │ │ D1 │ │ D2 │ │ D3 │ │ P0 │ │ P1 │ ← Object A (4+2) │
│ │Disk0│ │Disk1│ │Disk2│ │Disk3│ │Disk4│ │Disk5│ │
│ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ │
│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │
│ │ D0 │ │ D1 │ │ D2 │ │ D3 │ │ P0 │ │ P1 │ ← Object B (4+2) │
│ │Disk6│ │Disk7│ │Disk8│ │Disk9│ │Dk10│ │Dk11│ │
│ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Configuration (one OSD per drive):
iolakectl pool create p42 --ec-type 0 --ec-k 4 --ec-m 2 --failure-domain node
| Metric | Value |
|---|---|
| Storage efficiency | 66% (4/6) |
| Fault tolerance | 2 disk failures |
| Min disks | 6 |
| Read performance | Parallel from 4 disks |
| Repair cost | Read 4 shards, write 1 |
Option 1B: EC 8+4 (Maximum Fault Tolerance)¶
Configuration (one OSD per drive):
iolakectl pool create p84 --ec-type 0 --ec-k 8 --ec-m 4 --failure-domain node
| Metric | Value |
|---|---|
| Storage efficiency | 66% (8/12) |
| Fault tolerance | 4 disk failures |
| Min disks | 12 |
| Read performance | Parallel from 8 disks |
| Repair cost | Read 8 shards, write 1 |
Option 1C: Replication 3-way (Simple, Lower Efficiency)¶
Configuration (one OSD per drive):
iolakectl pool create r3 --ec-type 2 --replication-count 3 --failure-domain node
| Metric | Value |
|---|---|
| Storage efficiency | 33% (1/3) |
| Fault tolerance | 2 disk failures |
| Min disks | 3 |
| Read performance | Single disk (can load-balance) |
| Repair cost | Read 1, write 1 |
When to use replication: - Small objects (< 1MB) where EC overhead dominates - Random read workloads where single-disk latency matters - Simplest operational model
Scenario 2: Three Nodes (12 Disks Each)¶
Hardware: 3 servers, each with 12 disks (36 total disks)
Failure Domain Options: - Node-level: Survive 1 complete node failure - Disk-level: Survive multiple disk failures across nodes
Option 2A: EC 6+3 Node-Level (Recommended)¶
┌─────────────────────────────────────────────────────────────────────────┐
│ Node 0 (12 disks) │ Node 1 (12 disks) │ Node 2 (12 disks) │
│ ┌────┐┌────┐┌────┐ │ ┌────┐┌────┐┌────┐ │ ┌────┐┌────┐┌────┐ │
│ │ D0 ││ D1 ││ D2 │ │ │ D3 ││ D4 ││ D5 │ │ │ P0 ││ P1 ││ P2 │ │
│ └────┘└────┘└────┘ │ └────┘└────┘└────┘ │ └────┘└────┘└────┘ │
│ │ │ │
│ Each node gets 3 │ Each node gets 3 │ Each node gets 3 │
│ shards per object │ shards per object │ shards per object │
└─────────────────────────────────────────────────────────────────────────┘
Configuration (3 shards on each of 3 hosts):
iolakectl pool create p63 --ec-type 0 --ec-k 6 --ec-m 3 \
--failure-domain host --spread-domains 3 --per-domain 3
| Metric | Value |
|---|---|
| Storage efficiency | 66% (6/9) |
| Fault tolerance | 1 node OR 3 disks |
| Min nodes | 3 |
| Shards per node | 3 |
| Read performance | Parallel from 2+ nodes |
| Repair cost | Read 6 shards (cross-node) |
Math: With 3 nodes and k=6, m=3: - 9 shards ÷ 3 nodes = 3 shards per node - Node failure loses 3 shards = exactly m - Can reconstruct from remaining 6 shards ✓
Option 2B: EC 4+2 Node-Level (Fewer Shards)¶
┌─────────────────────────────────────────────────────────────────────────┐
│ Node 0 (12 disks) │ Node 1 (12 disks) │ Node 2 (12 disks) │
│ ┌────┐┌────┐ │ ┌────┐┌────┐ │ ┌────┐┌────┐ │
│ │ D0 ││ D1 │ │ │ D2 ││ D3 │ │ │ P0 ││ P1 │ │
│ └────┘└────┘ │ └────┘└────┘ │ └────┘└────┘ │
│ │ │ │
│ 2 shards per node │ 2 shards per node │ 2 shards per node │
└─────────────────────────────────────────────────────────────────────────┘
Configuration (2 shards on each of 3 hosts):
iolakectl pool create p42 --ec-type 0 --ec-k 4 --ec-m 2 \
--failure-domain host --spread-domains 3 --per-domain 2
| Metric | Value |
|---|---|
| Storage efficiency | 66% (4/6) |
| Fault tolerance | 1 node OR 2 disks |
| Min nodes | 3 |
| Shards per node | 2 |
Option 2C: Replication 3-way (Maximum Availability)¶
┌─────────────────────────────────────────────────────────────────────────┐
│ Node 0 │ Node 1 │ Node 2 │
│ ┌────────────────┐ │ ┌────────────────┐ │ ┌────────────────┐ │
│ │ Full Copy │ │ │ Full Copy │ │ │ Full Copy │ │
│ │ (Primary) │ │ │ (Replica 1) │ │ │ (Replica 2) │ │
│ └────────────────┘ │ └────────────────┘ │ └────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Configuration:
iolakectl pool create r3 --ec-type 2 --replication-count 3 --failure-domain host
| Metric | Value |
|---|---|
| Storage efficiency | 33% (1/3) |
| Fault tolerance | 2 node failures |
| Read performance | Any single node (load-balance) |
| Write performance | Must write to all 3 nodes |
Use replication when: - Data durability is paramount (financial, medical) - Random read latency is critical - Simple operational model preferred over efficiency
Scenario 3: Six+ Nodes (Multi-Rack with LRC)¶
Hardware: 6+ servers across multiple racks
Failure Domain: Rack-level: each local group in a rack of its own, so a lost shard is repaired inside its rack (a whole rack lost is not survived: see below)
Option 3A: LRC 6+2+2 (Recommended for 6+ Nodes)¶
LRC (Locally Repairable Codes) adds local parity within groups for faster single-failure repair.
┌─────────────────────────────────────────────────────────────────────────┐
│ LRC 6+2+2 │
│ │
│ ┌──────────────────────────────┐ ┌──────────────────────────────┐ │
│ │ Local Group 0 │ │ Local Group 1 │ │
│ │ ┌────┐┌────┐┌────┐┌────┐ │ │ ┌────┐┌────┐┌────┐┌────┐ │ │
│ │ │ D0 ││ D1 ││ D2 ││ LP0│ │ │ │ D3 ││ D4 ││ D5 ││ LP1│ │ │
│ │ │Rk0 ││Rk0 ││Rk0 ││Rk0 │ │ │ │Rk1 ││Rk1 ││Rk1 ││Rk1 │ │ │
│ │ └────┘└────┘└────┘└────┘ │ │ └────┘└────┘└────┘└────┘ │ │
│ └──────────────────────────────┘ └──────────────────────────────┘ │
│ │
│ ┌──────────────────────────────┐ │
│ │ Global Parity │ │
│ │ ┌────┐┌────┐ │ │
│ │ │GP0 ││GP1 │ │ GP = Reed-Solomon over all data │
│ │ │Rk2 ││Rk2 │ │ │
│ │ └────┘└────┘ │ │
│ └──────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Configuration (each local group in a rack of its own):
iolakectl pool create lrc622 --ec-type 1 --ec-k 6 \
--ec-local-parity 2 --ec-global-parity 2 \
--failure-domain rack --lrc-groups-per-domain
# local group size = k / local parity = 3
| Metric | Value |
|---|---|
| Storage efficiency | 60% (6/10) |
| Total shards | 10 (6 data + 2 LP + 2 GP) |
| Fault tolerance | Any 2 disks; most losses of 3 (what LRC survives). Not a rack holding a whole group (below) |
| Local repair | Read 3 shards (within rack) |
| Global repair | Read 6 shards (cross-rack) |
LRC Repair Scenarios:
| Failure Type | Repair Method | Network Cost |
|---|---|---|
| 1 data disk | XOR with local parity | Read 3 (local) |
| 1 local parity disk | XOR with data shards | Read 3 (local) |
| 2 disks same group | Reed-Solomon global | Read 6 (cross-rack) |
| 1 disk + 1 LP (diff) | Local + Local | Read 6 (2×3 local) |
| Full rack failure | Not survived with a group per rack: 3 data shards and the group's local parity lost, more than the 2 global parity rebuild | — |
Option 3B: LRC 10+2+2 (Higher Efficiency)¶
Configuration:
iolakectl pool create lrc1022 --ec-type 1 --ec-k 10 \
--ec-local-parity 2 --ec-global-parity 2 \
--failure-domain rack --lrc-groups-per-domain
| Metric | Value |
|---|---|
| Storage efficiency | 71% (10/14) |
| Total shards | 14 (10 data + 2 LP + 2 GP) |
| Min racks | 4 (5+1 per local group + GP) |
| Local repair | Read 5 shards (within rack) |
A pool keeps its losses decodable by the global parity: with each local group in a rack of its own, a rack's loss is survived only when a group has no more data shards than the global parity (meta warns when a pool's rule doesn't survive a whole domain lost). Neither 6+2+2 nor 10+2+2 does; for rack loss, spread an MDS profile over the racks. An LRC that keeps a rack's loss inside the rack is iolake#37.
Choosing Between EC and Replication¶
┌─────────────────────────────────────────┐
│ Decision Flow │
└─────────────────────────────────────────┘
│
┌─────────────────┴─────────────────┐
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Small objects │ │ Large objects │
│ (< 1 MB) │ │ (> 1 MB) │
└──────────┬──────────┘ └──────────┬──────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Replication 3-way │ │ Single DC? │
│ (EC overhead high) │ └──────────┬──────────┘
└─────────────────────┘ │
┌────────────────┴────────────────┐
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Yes: EC within DC │ │ No: Replicate │
│ (4+2 or 6+3) │ │ across DCs │
└──────────┬──────────┘ └─────────────────────┘
│
┌──────────┴──────────┐
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ 6+ nodes, racks? │ │ 3 nodes? │
│ → LRC 6+2+2 │ │ → EC 6+3 or 4+2 │
└─────────────────────┘ └─────────────────────┘
Configuration Examples¶
OSDs¶
Each OSD's osd.toml names its drive and its failure domain; OSDs on one
host share a host name (the hostname, by default), and set rack (and
datacenter) where a pool spreads over those:
# /etc/objectio/osd.toml (one per OSD; one OSD per drive)
[osd]
listen = "0.0.0.0:9200"
advertise_addr = "http://node1.internal:9200"
meta_endpoint = "http://meta1.internal:9100,http://meta2.internal:9100,http://meta3.internal:9100"
[osd.failure_domain]
datacenter = "dc1"
rack = "rack-01" # a different rack per node, for a rack pool
[storage]
disks = ["/dev/disk/by-id/nvme-…"]
data_dir = "/var/lib/objectio/osd-0"
Several OSDs on one host each need their own port, data_dir and drive.
Pools¶
The default pool's code is meta's --ec-k/--ec-m (or
--replication), spread over hosts. Any other layout above is a pool of
its own, made with iolakectl pool create as shown in each scenario, and
buckets are put in it when they are created. The gateway's own
--ec-k/--ec-m don't choose where a bucket's data goes.
Capacity Planning¶
Usable Capacity Formula¶
Usable = Raw × (k / (k + m)) # For MDS
Usable = Raw × (k / (k + l + g)) # For LRC
Example: 3 Nodes × 12 Disks × 10TB¶
| Config | Raw Capacity | Efficiency | Usable Capacity |
|---|---|---|---|
| Replication 3 | 360 TB | 33% | 120 TB |
| EC 4+2 | 360 TB | 66% | 240 TB |
| EC 6+3 | 360 TB | 66% | 240 TB |
| LRC 6+2+2 | 360 TB | 60% | 216 TB |
IOPS Considerations¶
| Config | Read IOPS (per object) | Write IOPS (per object) |
|---|---|---|
| Replication 3 | 1 disk (can load-balance) | 3 disks (all replicas) |
| EC 4+2 | 4 disks (parallel) | 6 disks (parallel) |
| EC 6+3 | 6 disks (parallel) | 9 disks (parallel) |
| LRC 6+2+2 | 6 disks (parallel) | 10 disks (parallel) |
Migration Paths¶
Starting Small, Growing Large¶
Phase 1: Single node, 6 disks
└── EC 4+2 (disk-level)
│
▼
Phase 2: 3 nodes, 12 disks each
└── EC 6+3 (node-level)
│
▼
Phase 3: 6+ nodes across racks
└── LRC 6+2+2 (rack-level)
Growing: a pool's code and its number of placement groups are fixed when it is made. OSDs added to a pool's failure domains take a share of its placement groups as the balancer and recovery move them. To move to a wider code (4+2 to 6+3, or to LRC), make a new pool and put new buckets in it; existing buckets stay in their pool. Copy a bucket across with any S3 tool if it must move.
Summary Recommendations¶
| Deployment Size | Recommended Config | Why |
|---|---|---|
| Dev/Test | Single node, EC 4+2 | Simple, efficient, survives disk failures |
| Small Prod | 3 nodes, EC 6+3 | Survives node failure, good efficiency |
| Medium Prod | 6 nodes, LRC 6+2+2 | Fast local repair, rack-level tolerance |
| Large Prod | 10+ nodes, LRC 10+2+2 | Best efficiency (71%), scalable |
| Critical | 3-way replication | Maximum durability, simple operations |
See Also¶
- Data Protection - Deep dive into erasure coding math
- Placement - Placement groups and failure domains
- Failure Recovery - Repair and rebalancing procedures