Deployment Topologies¶
IOLake supports various deployment configurations. For worked scenarios, configuration examples and capacity planning, see deployment-options.md.
Topology 1: Single-Node Multi-Disk¶
Best for development, edge, and small deployments.
┌─────────────────────────────────────────────────────┐
│ Single Server │
│ │
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │ Disk 1 │ │ Disk 2 │ │ Disk 3 │ │ Disk 4 │ ... │
│ │ Shard1 │ │ Shard2 │ │ Shard3 │ │ Shard4 │ │
│ └────────┘ └────────┘ └────────┘ └────────┘ │
│ │
│ EC 4+2: Spread 6 shards across 6+ disks │
│ │
│ Components: │
│ • iolake-gateway (port 9000) │
│ • iolake-meta (embedded, single-node mode) │
│ • iolake-osd (manages all disks) │
└─────────────────────────────────────────────────────┘
Characteristics: - Survives: Disk failures (up to m disks) - Does NOT survive: Server failure - Storage efficiency: 67% (4+2)
Example: one host, six drives, one OSD per drive, each with its own
osd.toml (see Linux), and meta started with
--ec-k 4 --ec-m 2. With fewer hosts than shards, the default pool
spreads each stripe over OSDs instead of hosts. For a try-out in one
process: iolake-aio --osds 6 --ec-k 4 --ec-m 2.
Topology 2: Multi-Node Single-DC (Rack Aware)¶
Production single-datacenter deployment.
┌─────────────────────────────────────────────────────────────────────┐
│ Datacenter │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Rack A │ │ Rack B │ │ Rack C │ │
│ │ ┌───────────┐ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │
│ │ │ Node 1 │ │ │ │ Node 3 │ │ │ │ Node 5 │ │ │
│ │ │ D1 D2 D3 │ │ │ │ D1 D2 D3 │ │ │ │ D1 D2 D3 │ │ │
│ │ │ Shards: │ │ │ │ Shards: │ │ │ │ Shards: │ │ │
│ │ │ 1,4,7,10 │ │ │ │ 2,5,8,11 │ │ │ │ 3,6,9,12 │ │ │
│ │ └───────────┘ │ │ └───────────┘ │ │ └───────────┘ │ │
│ │ ┌───────────┐ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │
│ │ │ Node 2 │ │ │ │ Node 4 │ │ │ │ Node 6 │ │ │
│ │ │ D1 D2 D3 │ │ │ │ D1 D2 D3 │ │ │ │ D1 D2 D3 │ │ │
│ │ └───────────┘ │ │ └───────────┘ │ │ └───────────┘ │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
│ EC 8+4: Shards distributed across racks │
│ Metadata: 3-node Raft cluster (one per rack) │
└─────────────────────────────────────────────────────────────────────┘
Characteristics: - Survives: Multiple disk, node, and rack failures - Does NOT survive: DC-wide failure - Typical setup: 3+ racks, 6+ nodes
Placement: each OSD names its rack in osd.toml
([osd.failure_domain] rack = "rack-a"; the host defaults to the
machine's hostname). The default pool spreads over hosts. For racks,
create a pool with a rack failure domain and put buckets in it:
# 8+4 over 3 racks, at most 4 shards of a stripe in one rack
iolakectl pool create racked --ec-type 0 --ec-k 8 --ec-m 4 \
--failure-domain rack --spread-domains 3 --per-domain 4
iolakectl bucket create analytics --pool racked
Losing a rack then loses at most 4 shards, which 8+4 survives. Without
--spread-domains/--per-domain a pool puts one shard per domain, so
8+4 would need 12 racks. A bucket's pool is fixed when it is created
(x-iolake-pool on CreateBucket, or the tenant's default pool).
Topology 3: Multi-DC / Multi-AZ¶
Geo-distributed deployment for disaster recovery.
┌───────────────────────┐ ┌───────────────────────┐ ┌───────────────────────┐
│ Datacenter 1 │ │ Datacenter 2 │ │ Datacenter 3 │
│ (Primary) │ │ (Secondary) │ │ (DR) │
│ │ │ │ │ │
│ ┌─────────────────┐ │ │ ┌─────────────────┐ │ │ ┌─────────────────┐ │
│ │ Full Copy 1 │ │ │ │ Full Copy 2 │ │ │ │ Full Copy 3 │ │
│ │ (EC within) │◄─┼────┼─►│ (EC within) │◄─┼────┼─►│ (EC within) │ │
│ └─────────────────┘ │ │ └─────────────────┘ │ │ └─────────────────┘ │
│ │ │ │ │ │ │ │ │
│ ┌────┴────┐ │ │ ┌────┴────┐ │ │ ┌────┴────┐ │
│ │Racks/ │ │ │ │Racks/ │ │ │ │Racks/ │ │
│ │Nodes │ │ │ │Nodes │ │ │ │Nodes │ │
│ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │
│ │ │ │ │ │
│ Metadata: Local Raft │ │ Metadata: Local Raft │ │ Metadata: Local Raft │
└───────────────────────┘ └───────────────────────┘ └───────────────────────┘
│ │ │
└────────────────────────────┼────────────────────────────┘
│
Async Replication
Each site is a cluster of its own, with its own meta Raft group; nothing is shared between them.
Data flow: 1. A write arrives at a DC1 gateway and is erasure-coded within DC1. 2. Bucket replication sends each new version, asynchronously, to a bucket on the DC2 (and DC3) cluster. 3. Clients read from the cluster they are pointed at.
Characteristics:
- Survives: a complete DC/AZ, losing what hadn't replicated yet
(iolake_replication_lag_seconds).
- Use case: geo-distributed, disaster recovery, compliance
Configuration: per bucket, on the source cluster: a replication target naming the remote cluster's bucket, and the bucket's replication rules (replication).
Comparison¶
| Topology | Failure Tolerance | Complexity | Use Case |
|---|---|---|---|
| Single-node | Disk only | Low | Dev, edge |
| Multi-node | Disk, node, rack | Medium | Production |
| Multi-DC | Disk, node, rack, DC | High | Multi-site production |
Choosing a Topology¶
Single-Node Multi-Disk¶
Choose if: - Development or testing - Edge deployment with limited hardware - Cost-sensitive deployments - Acceptable single-node failure risk
Multi-Node Single-DC¶
Choose if: - Production workloads - Node and rack fault tolerance required - Single datacenter deployment - No geo-redundancy needs
Multi-DC / Multi-AZ¶
Choose if: - Business-critical data - Regulatory compliance (data locality) - Disaster recovery requirements - Global distribution needed