# Set up a cluster

Configure three Haematite-backed nodes with explicit membership, shards and peer addresses.

Aion can replicate its Haematite store across nodes. Each node has its own home, data directory and listeners. This guide configures a new three-node cluster using a newer main build.

## Prepare the nodes

Use the same selected Aion build on all three machines. Reserve stable private addresses and distinct node names. Keep replication traffic on the controlled cluster network. Configure [authentication](/aion/operations/auth-and-lys/) for client access.

Start from the config generated by that main build. Retain its runtime, outbox, websocket, observability and supervision sections. Set the worker listener to each node’s reachable address when remote workers need it.

- `Membership`: Use the same full members list on every node. Quorum uses the full set, including unreachable members. Three members need a majority of two.
- `Shard count`: Set the same immutable count before creating the stores. This example uses four fresh shards across three nodes. Existing stores retain their on-disk count.
- `Ownership`: Explicit owned_shards splits initial ownership. An empty list means all shards. Each peer entry repeats that peer’s assigned shards for automatic failover.
- `Addresses`: bind_address and peer address carry replication. grpc_address is the peer client API used for forwarding. Each node also needs its own HTTP and worker listener addresses.
- `Cancellation`: Choose positive relay base and maximum milliseconds and multiplier at least one. The maximum must be at least the base. The three values are required with a cluster section.
- `Implementation`: crates/aion-server/src/config/sections.rs:88–134 and 234–331 define membership, ownership, forwarding and failover configuration.


## Assign the shards

Use these example sections for fresh stores. Replace the private addresses and paths for your deployment. Preserve the identical membership list and complementary shard assignment on every node.

**node-1.toml — merge into that node’s main config**

```toml
[server]
listen_address = "10.20.0.1:8080"
grpc_addresses = ["10.20.0.1:50051"]

[store]
backend = "haematite"
data_dir = "/srv/aion/node-1/data"
shard_count = 4
owned_shards = [0, 1]
node_cache_budget = "unlimited"

[store.cluster]
node_id = "node-1@10.20.0.1"
bind_address = "10.20.0.1:7000"
members = ["node-1@10.20.0.1", "node-2@10.20.0.2", "node-3@10.20.0.3"]
cancel_relay_backoff_base_ms = 100
cancel_relay_backoff_multiplier = 2
cancel_relay_backoff_max_ms = 5000

[[store.cluster.peers]]
name = "node-2@10.20.0.2"
address = "10.20.0.2:7000"
grpc_address = "10.20.0.2:50051"
owned_shards = [2]

[[store.cluster.peers]]
name = "node-3@10.20.0.3"
address = "10.20.0.3:7000"
grpc_address = "10.20.0.3:50051"
owned_shards = [3]
```


**node-2.toml — merge into that node’s main config**

```toml
[server]
listen_address = "10.20.0.2:8080"
grpc_addresses = ["10.20.0.2:50051"]

[store]
backend = "haematite"
data_dir = "/srv/aion/node-2/data"
shard_count = 4
owned_shards = [2]
node_cache_budget = "unlimited"

[store.cluster]
node_id = "node-2@10.20.0.2"
bind_address = "10.20.0.2:7000"
members = ["node-1@10.20.0.1", "node-2@10.20.0.2", "node-3@10.20.0.3"]
cancel_relay_backoff_base_ms = 100
cancel_relay_backoff_multiplier = 2
cancel_relay_backoff_max_ms = 5000

[[store.cluster.peers]]
name = "node-1@10.20.0.1"
address = "10.20.0.1:7000"
grpc_address = "10.20.0.1:50051"
owned_shards = [0, 1]

[[store.cluster.peers]]
name = "node-3@10.20.0.3"
address = "10.20.0.3:7000"
grpc_address = "10.20.0.3:50051"
owned_shards = [3]
```


**node-3.toml — merge into that node’s main config**

```toml
[server]
listen_address = "10.20.0.3:8080"
grpc_addresses = ["10.20.0.3:50051"]

[store]
backend = "haematite"
data_dir = "/srv/aion/node-3/data"
shard_count = 4
owned_shards = [3]
node_cache_budget = "unlimited"

[store.cluster]
node_id = "node-3@10.20.0.3"
bind_address = "10.20.0.3:7000"
members = ["node-1@10.20.0.1", "node-2@10.20.0.2", "node-3@10.20.0.3"]
cancel_relay_backoff_base_ms = 100
cancel_relay_backoff_multiplier = 2
cancel_relay_backoff_max_ms = 5000

[[store.cluster.peers]]
name = "node-1@10.20.0.1"
address = "10.20.0.1:7000"
grpc_address = "10.20.0.1:50051"
owned_shards = [0, 1]

[[store.cluster.peers]]
name = "node-2@10.20.0.2"
address = "10.20.0.2:7000"
grpc_address = "10.20.0.2:50051"
owned_shards = [2]
```


Each node stores data in its own directory. Preserve existing data and its recorded shard count when expanding an existing installation; plan the ownership change and backup before changing membership.

## Start the prepared cluster

Run the foreground server on each node using that node’s config and home. This is the command for node one; use the matching node number on the other two machines.

**Terminal — on each prepared node**

```bash
AION_HOME=/srv/aion/node-1/home aion server --config /etc/aion/node-1.toml
```


Wait for quorum and readiness. Read each node’s build identity, shard ownership and recovery log. Deploy required packages on the nodes that will serve them and retain those versions for recovery.

## Attach clients and workers

Give clients a reachable gRPC or HTTP listener. Declare peer gRPC addresses to enable forwarding of operations to the owner. Liminal workers can receive several server addresses as ordered reconnect candidates.

Read [worker commands](/aion/reference/cli-workers/) and [configuration fields](/aion/reference/configuration-engine/) for listener and failover settings. Exercise loss and restoration of a node on the deployment’s storage before relying on automatic failover.

On main, crates/aion-server/src/config/sections.rs:88–134 and :234–331 define these settings. Cluster cancellation relay and recovery use the paths described in [workflow cancellation](/aion/operations/activity-control/). Verify failover and recovery on the storage and network used by your deployment.