Operate · 01

Run a three-node cluster.

Three nodes survive one failure. The leader runs writes, a majority confirms them, and any node can answer reads.

Start three nodes

Stop any single-node server first, since it holds port 7101. Run each server in its own terminal with the same operator token. Each node needs its own ID, address and data directory.

Each server terminal · same token
export FLOWER_ADMIN_TOKEN='replace-with-a-shared-operator-secret'
# Run one of these commands per terminal:
./target/release/flower --id 1 --listen 127.0.0.1:7101 --data .flower/cluster/node1
./target/release/flower --id 2 --listen 127.0.0.1:7102 --data .flower/cluster/node2
./target/release/flower --id 3 --listen 127.0.0.1:7103 --data .flower/cluster/node3
Operator terminal · bootstrap once
export FLOWER_ADMIN_TOKEN='replace-with-a-shared-operator-secret'
node sdk/cli.ts init \
  --members 1=127.0.0.1:7101,2=127.0.0.1:7102,3=127.0.0.1:7103

curl -fsS -H "Authorization: Bearer $FLOWER_ADMIN_TOKEN" \
  http://127.0.0.1:7101/raft/metrics

Point clients at any node. Writes and deployments go to the leader for you; queries and watches run on the node you called.

Select any reachable member
export FLOWER_URL='http://127.0.0.1:7102'
node sdk/cli.ts deploy examples/orders.ts
# Or pass --url http://127.0.0.1:7102 on an individual command.
  • Separate hosts: use addresses the other nodes can reach. --listen is the socket to bind; --advertise HOST:PORT is the address peers use, if different. Member addresses have no http://.
  • If your entry node dies, switch clients to another node. Retrying calls and reconnecting watches is up to your application.
  • Failover usually starts within about 0.5 s (50 ms heartbeat, 150–300 ms election timeout, 300 ms leader lease). On slow networks or disks, raise FLOWER_RAFT_HEARTBEAT_MS, FLOWER_RAFT_ELECTION_MIN_MS and FLOWER_RAFT_ELECTION_MAX_MS. Use the same integer milliseconds on every node, with heartbeat < min < max. Values that are too short cause needless elections under load.
  • Node-to-node traffic uses HTTP/2 on the same ports. It is not encrypted unless you enable TLS (see below).

What an acknowledged write means

A mutation reply means a quorum stored the change durably and the node you called applied it. Crashes after that don't lose it.

  • One node can fail. Without a quorum, writes and fresh reads stop. Replica-local reads keep serving local data, which can be stale with no bound.
  • A timed-out mutation may still have committed. Retry with the same request ID.
  • Restart a stopped node with the same ID and data directory. It catches up on its own. Never run init on it again or reuse its directory under another ID. While it catches up, even its replica-local reads can be unavailable.
  • Live data is held in memory. Redb application tables hold the durable state for recovery. Raft checkpoints write metadata only; full images are encoded when a replica needs a snapshot transfer. Snapshot status is in GET /admin/resources; see the snapshot settings to tune when they run.

Flower is unreleased. APIs and storage and wire formats can change with no migration path. You can't downgrade a data directory, and some upgrades can't be rolled one node at a time: old and new builds refuse to talk, so upgrade all nodes together.

Day-to-day operations

  1. Check the process. GET /health only says the process is up. It doesn't prove quorum.
  2. Check consensus. GET /raft/metrics (operator token) shows the leader, members and replication progress. Read server logs during elections or storage errors.
  3. Check load. GET /admin/resources (operator token) shows worker pools, memory budgets and snapshots. Tune them with the capacity settings.
  4. Deploy deliberately. A deployment switches code and recomputed values in one step; a rejected one leaves the old version running. For large rebuilds, use staged deployment and page tuning. Data migrations are explicit.
  5. Keep node data safe. Give each node its own persistent directory. Add nodes, remove nodes and upgrade with the membership API and rolling-upgrade procedure, one node at a time.

Before production: enable TLS with the three FLOWER_TLS_*_FILE settings, and set a separate FLOWER_PEER_TOKEN for node-to-node traffic (it falls back to FLOWER_ADMIN_TOKEN otherwise). Only trusted operators and nodes should reach admin endpoints. Flower has no identity provider; your authorize hooks decide who callers are. A compromised node compromises the cluster. See the TLS and token rotation guide.

Move partitions between groups

To spread load, move named partitions to other Raft groups. A partition's data, code, timers, leases and retry history move with it.

  • Create and register the new groups first, then call client.resize(groupIds). Flower moves one partition at a time.
  • A partition keeps serving while it's copied and pauses only for the final cutover.
  • Groups still share the machine's resources. Balancing is by partition count.
  • A failed move resumes when its groups recover.
  • Existing composite tenant keys are not movable partitions.

See the partition API for the details.