ablative docs

Go to

↑↓ move↵ openesc close
 Stack Notebook ablative.com.au

Set up a cluster

Configure three Haematite-backed nodes with explicit membership, shards and peer addresses.

Open .md

Prepare the nodes

Use the same selected Aion build on all three machines. Reserve stable private addresses and distinct node names. Keep replication traffic on the controlled cluster network. Configure authentication for client access.

Start from the config generated by that main build. Retain its runtime, outbox, websocket, observability and supervision sections. Set the worker listener to each node’s reachable address when remote workers need it.

Membership
Use the same full members list on every node. Quorum uses the full set, including unreachable members. Three members need a majority of two.
Shard count
Set the same immutable count before creating the stores. This example uses four fresh shards across three nodes. Existing stores retain their on-disk count.
Ownership
Explicit owned_shards splits initial ownership. An empty list means all shards. Each peer entry repeats that peer’s assigned shards for automatic failover.
Addresses
bind_address and peer address carry replication. grpc_address is the peer client API used for forwarding. Each node also needs its own HTTP and worker listener addresses.
Cancellation
Choose positive relay base and maximum milliseconds and multiplier at least one. The maximum must be at least the base. The three values are required with a cluster section.
Implementation
crates/aion-server/src/config/sections.rs:88–134 and 234–331 define membership, ownership, forwarding and failover configuration.

Assign the shards

Use these example sections for fresh stores. Replace the private addresses and paths for your deployment. Preserve the identical membership list and complementary shard assignment on every node.

[server]
listen_address = "10.20.0.1:8080"
grpc_addresses = ["10.20.0.1:50051"]

[store]
backend = "haematite"
data_dir = "/srv/aion/node-1/data"
shard_count = 4
owned_shards = [0, 1]
node_cache_budget = "unlimited"

[store.cluster]
node_id = "node-1@10.20.0.1"
bind_address = "10.20.0.1:7000"
members = ["node-1@10.20.0.1", "node-2@10.20.0.2", "node-3@10.20.0.3"]
cancel_relay_backoff_base_ms = 100
cancel_relay_backoff_multiplier = 2
cancel_relay_backoff_max_ms = 5000

[[store.cluster.peers]]
name = "node-2@10.20.0.2"
address = "10.20.0.2:7000"
grpc_address = "10.20.0.2:50051"
owned_shards = [2]

[[store.cluster.peers]]
name = "node-3@10.20.0.3"
address = "10.20.0.3:7000"
grpc_address = "10.20.0.3:50051"
owned_shards = [3]
[server]
listen_address = "10.20.0.2:8080"
grpc_addresses = ["10.20.0.2:50051"]

[store]
backend = "haematite"
data_dir = "/srv/aion/node-2/data"
shard_count = 4
owned_shards = [2]
node_cache_budget = "unlimited"

[store.cluster]
node_id = "node-2@10.20.0.2"
bind_address = "10.20.0.2:7000"
members = ["node-1@10.20.0.1", "node-2@10.20.0.2", "node-3@10.20.0.3"]
cancel_relay_backoff_base_ms = 100
cancel_relay_backoff_multiplier = 2
cancel_relay_backoff_max_ms = 5000

[[store.cluster.peers]]
name = "node-1@10.20.0.1"
address = "10.20.0.1:7000"
grpc_address = "10.20.0.1:50051"
owned_shards = [0, 1]

[[store.cluster.peers]]
name = "node-3@10.20.0.3"
address = "10.20.0.3:7000"
grpc_address = "10.20.0.3:50051"
owned_shards = [3]
[server]
listen_address = "10.20.0.3:8080"
grpc_addresses = ["10.20.0.3:50051"]

[store]
backend = "haematite"
data_dir = "/srv/aion/node-3/data"
shard_count = 4
owned_shards = [3]
node_cache_budget = "unlimited"

[store.cluster]
node_id = "node-3@10.20.0.3"
bind_address = "10.20.0.3:7000"
members = ["node-1@10.20.0.1", "node-2@10.20.0.2", "node-3@10.20.0.3"]
cancel_relay_backoff_base_ms = 100
cancel_relay_backoff_multiplier = 2
cancel_relay_backoff_max_ms = 5000

[[store.cluster.peers]]
name = "node-1@10.20.0.1"
address = "10.20.0.1:7000"
grpc_address = "10.20.0.1:50051"
owned_shards = [0, 1]

[[store.cluster.peers]]
name = "node-2@10.20.0.2"
address = "10.20.0.2:7000"
grpc_address = "10.20.0.2:50051"
owned_shards = [2]

Each node stores data in its own directory. Preserve existing data and its recorded shard count when expanding an existing installation; plan the ownership change and backup before changing membership.

Start the prepared cluster

Run the foreground server on each node using that node’s config and home. This is the command for node one; use the matching node number on the other two machines.

AION_HOME=/srv/aion/node-1/home aion server --config /etc/aion/node-1.toml

Wait for quorum and readiness. Read each node’s build identity, shard ownership and recovery log. Deploy required packages on the nodes that will serve them and retain those versions for recovery.

Attach clients and workers

Give clients a reachable gRPC or HTTP listener. Declare peer gRPC addresses to enable forwarding of operations to the owner. Liminal workers can receive several server addresses as ordered reconnect candidates.

Read worker commands and configuration fields for listener and failover settings. Exercise loss and restoration of a node on the deployment’s storage before relying on automatic failover.

On main, crates/aion-server/src/config/sections.rs:88–134 and :234–331 define these settings. Cluster cancellation relay and recovery use the paths described in workflow cancellation. Verify failover and recovery on the storage and network used by your deployment.