Flower / Field notes

Across the garden.

Independent kitchens. Parallel Raft groups. One measured rush.
8 groups × 3 replicas · 16 tenants · 4 stores per tenant.

Passed

2026-09-24T22:30:32.550Z → 2026-09-24T22:31:32.688Z

Global customer goodput89,127.3 / sSuccessful customer calls divided by the union load window.
Customer p99247.2 msMerged customer histogram buckets; never averaged percentiles.
Successful customer calls5,359,9363,751,020 reads · 1,608,916 mutations · 0 failures
Independent auditsPassed768 orders · 1,914 pizzas · 160,726 replay checks
Replica-local customer reads · lag is allowed. Each read uses one coherent locally applied committed replica snapshot without a per-read quorum fence. Startup recovery can first require quorum. State, code, and aliases may lag without a bound; requests across replicas can go backward in revision. This policy must not be confused with fresh-read performance. Mutations and the final per-group audit retain their fresh correctness requirements.

What the total means

Successful primary customer logical calls across all groups divided by the union of their synchronized load intervals, including in-flight tails. Excludes retries, explicit replays, workers, errors, warmup and drain. Percentiles merge bucket counts; they are not averages of group percentiles.

Static tenant-to-group assignment. Each tenant's stores, queues and rankings stay within one independent Raft group. No cross-group transaction or global atomic snapshot in this workload.

Measured union window
60.14 s
Shared overlap
60.05 s
Start skew
1.0 ms
Actual successful read / mutation mix
70.0% / 30.0%
Correctness
Passed
Global and individual group goodputGlobal uses the union window. Each group uses its own measured interval; adding those rates is not the global measurement. Global: 89,127.3 successful customer calls/s; group-0: 11,247.8 successful customer calls/s; group-1: 11,129.2 successful customer calls/s; group-2: 11,149.1 successful customer calls/s; group-3: 11,095.6 successful customer calls/s; group-4: 11,147.1 successful customer calls/s; group-5: 11,202.8 successful customer calls/s; group-6: 11,051.6 successful customer calls/s; group-7: 11,164.7 successful customer calls/sGlobal89,127.3group-011,247.8group-111,129.2group-211,149.1group-311,095.6group-411,147.1group-511,202.8group-611,051.6group-711,164.7successful customer calls/s · linear scale
Global uses the union window. Each group uses its own measured interval; adding those rates is not the global measurement.

Every group keeps its own books.

Group outcomes. Open the child reports for per-node routing, full invariant checks, lifecycle timing, and diagnostics.
Raft groupPartitionGoodput / sCompletedFailedCustomer p99Fresh auditQuorum recoveryDetails
group-02 tenantstenant-0, tenant-111,247.8675,6820252.2 msPassed96 / 96 delivered617.6 msHTML · JSON
group-12 tenantstenant-2, tenant-311,129.2669,2400237.6 msPassed96 / 96 delivered647.9 msHTML · JSON
group-22 tenantstenant-4, tenant-511,149.1670,4820239.9 msPassed96 / 96 delivered567.8 msHTML · JSON
group-32 tenantstenant-6, tenant-711,095.6666,5300262.4 msPassed96 / 96 delivered635.4 msHTML · JSON
group-42 tenantstenant-8, tenant-911,147.1670,0010247.2 msPassed96 / 96 delivered555.7 msHTML · JSON
group-52 tenantstenant-10, tenant-1111,202.8673,1060244.8 msPassed96 / 96 delivered621.9 msHTML · JSON
group-62 tenantstenant-12, tenant-1311,051.6664,4070262.4 msPassed96 / 96 delivered568.6 msHTML · JSON
group-72 tenantstenant-14, tenant-1511,164.7670,4880230.6 msPassed96 / 96 delivered745.1 msHTML · JSON

Per-group p99 uses each group's customer histogram. Recovery is measured separately per injected crash; it is not a cluster-wide failover time.

The tail belongs to the customers.

Combined customer latencyAll groups' bucket counts are merged before extracting quantiles. These are logical customer calls, including retries until success or terminal failure. All p50: 6.6 ms; All p95: 161.2 ms; All p99: 247.2 ms; reads p50: 3.0 ms; reads p95: 32.2 ms; reads p99: 69.2 ms; mutations p50: 104.0 ms; mutations p95: 219.4 ms; mutations p99: 491.2 msAll p506.6All p95161.2All p99247.2reads p503.0reads p9532.2reads p9969.2mutations p50104.0mutations p95219.4mutations p99491.2ms · linear scale
All groups' bucket counts are merged before extracting quantiles. These are logical customer calls, including retries until success or terminal failure.
Combined customer populations; worker and explicit replay methods are excluded.
Customer populationSamplesp50p95p99Maximum
All customer calls5,359,9366.6 ms161.2 ms247.2 ms1,740.7 ms
Customer reads3,751,0203.0 ms32.2 ms69.2 ms1,740.7 ms
Customer mutations1,608,916104.0 ms219.4 ms491.2 ms940.6 ms

Nearest-rank logarithmic histogram upper bounds are approximate (within 1% over the configured normal bucket range). Empty populations stay unavailable. This report never averages p99 values or fabricates missing samples. Customer failures contribute latency observations but zero successful goodput.

Recovery and the full delivery cycle.

Leader failure under customer load

One row per recorded leader crash. Recovery and catch-up both start at the crash, not at each other.
GroupCrashTime into loadLeader before → afterServing quorumRestarted replica caught up
group-0130.01 s1 → 3617.6 ms4,319.7 ms
group-1130.04 s1 → 3647.9 ms4,808.2 ms
group-2130.03 s1 → 2567.8 ms4,260.5 ms
group-3130.02 s1 → 2635.4 ms4,885.9 ms
group-4130.04 s1 → 3555.7 ms4,332.5 ms
group-5130.03 s1 → 3621.9 ms4,879.4 ms
group-6130.01 s1 → 3568.6 ms4,314.8 ms
group-7130.02 s1 → 2745.1 ms4,957.8 ms

A serving quorum can resume before the killed replica rejoins. These polling observations are per group; they do not measure the time until every client succeeds. Missing milestones remain unavailable.

Orders, timers, and safety probes

Audited orders
768
Audited pizzas
1,914
Exact receipt replay checks
160,726
Expired-lease rejection checks
94
Configured baking delay
250.0 ms
Configured lease duration
2,000.0 ms
Delivery and timer sample counts include load and drain. Safety-probe counts cover the complete run.
GroupDeliveriesDelivery p50Delivery p95Delivery p99TimersLateness p95Lateness p99ReplaysStale leases
group-0963,629.5 ms6,528.4 ms6,669.0 ms961,110.0 ms1,110.0 ms20,21512
group-1963,157.5 ms5,350.3 ms5,977.0 ms961,238.0 ms1,238.0 ms19,9598
group-2963,157.5 ms5,969.2 ms7,363.0 ms961,045.0 ms1,045.0 ms20,2659
group-3963,702.4 ms6,089.2 ms7,619.0 ms961,315.4 ms1,317.0 ms20,09413
group-4963,852.8 ms6,593.7 ms8,253.0 ms961,187.0 ms1,187.0 ms19,95810
group-5963,739.5 ms6,028.9 ms8,116.0 ms961,226.9 ms1,549.0 ms19,94611
group-6963,702.4 ms6,659.6 ms7,647.0 ms96956.7 ms958.0 ms20,07217
group-7963,419.1 ms6,463.8 ms6,694.0 ms961,114.0 ms1,114.0 ms20,21714

Order-to-delivery includes baking, queueing, abandoned leases, and delivery. Timer lateness measures callback execution after its due time, excluding the configured baking delay. These per-group lifecycle percentiles are separate from customer request latency and are not averaged into a global percentile. Replay and expired-lease probes contribute to correctness checks, not customer goodput. Expected stale-lease rejections appear as HTTP errors in the child reports.

The garden shares a machine.

Server RSS sums add per-process sampled peaks; those peaks need not coincide. CPU is the sum of sampled per-node mean cores.
GroupServer peak sums, MiBNode peak, MiBSampled server coresNode load coresNode user CPUNode system CPUNode loop p99Rust driver peak, MiBRust driver coresDisk, MiB
group-01,303.2413.01.260.054,319.9 ms529.0 ms44.5 ms24.70.23534.1
group-11,256.8384.71.260.064,429.4 ms584.7 ms45.9 ms24.80.23531.1
group-21,260.6400.61.260.054,501.1 ms587.6 ms43.8 ms24.90.22534.1
group-31,263.2366.21.240.054,345.1 ms529.7 ms45.3 ms25.00.22536.8
group-41,246.5378.01.250.054,401.9 ms588.1 ms47.5 ms25.20.22536.3
group-51,320.7375.01.260.054,339.5 ms592.2 ms46.0 ms25.20.23534.1
group-61,230.2367.71.250.054,323.1 ms545.2 ms46.2 ms25.10.22537.1
group-71,329.1367.41.260.054,393.0 ms589.1 ms46.2 ms24.60.22537.1

Each child generator and every replica run on this host. Adding Raft groups adds writer lanes, replication work, drivers, memory, and disk contention; these results do not establish multi-host capacity or linear scaling. Node user/system CPU covers the whole run; Node load cores and optional Rust driver cores cover sampled load intervals. Both processes count toward driver cost. Sampling can miss short peaks; missing measurements are shown as unavailable.

Enough detail to repeat it.

CPU
Apple M5 Pro
OS / architecture
darwin 27.2.0 / arm64
Logical CPUs
18
Database processes
24
Generator/controller processes
16
Customer driver
rust
Node.js controller
v26.10.0
Driver SHA-256
1cf843cbf4e064d13a1d3597d9295c509325fee5c74b6eb2242ac7e98ece553d
Customer loops per group
512
Drone loops per group
4
Tenants per group / stores per tenant
2 / 4
Hot stores per tenant
1
Hot-set mixture weight
80.0%
Read endpoints
replicas
Customer read policy
replica-local
Application transport
HTTP/2 · pooled h2c
Requested load / warmup
60.0 s / 1.0 s
Order cap per group
96
Maximum drain
60.0 s
Initialization
static
Binary SHA-256
3b47cf25cfa2a278f6b96e9ab675c70694b1ad4408e6f09a18e2555c0939c9c9
Bundle hash
264f2718f2336ea92b850ec939282695609e4ac929a73006455bf212872473b0
Engine
quickjs

Hot-set weight is mixed with uniform store selection; it is not the final observed hot-store probability. Tenant selection is spread independently. Each tenant's stores and derived ranking remain in its assigned group. A busy tenant still shares one ordered writer lane; adding unrelated groups does not speed up that tenant's serial mutations.

Customer loops are closed-loop: each waits before issuing its next call. After the order cap, order choices become reads. Compare runs with the same workload, consistency, host, binary, and runtime settings. These are simultaneous independent-group measurements, not a globally atomic transaction or snapshot benchmark.

Runtime settings and complete workload options
{
  "runtime": {
    "engine": "quickjs",
    "settings": {
      "TOKIO_WORKER_THREADS": "2",
      "FLOWER_WRITER_BATCH_MODE": "adaptive",
      "FLOWER_WRITER_QUEUE_CAPACITY": "1024",
      "FLOWER_QUERY_WORKERS": "16",
      "FLOWER_PREPARATION_WORKERS": "16",
      "FLOWER_WRITER_PREPARATION_WORKERS": "1",
      "FLOWER_WRITER_BATCH_MS": "200",
      "FLOWER_OTEL_ENABLED": "0",
      "OTEL_SDK_DISABLED": "true"
    }
  },
  "environment": {
    "node": "v26.10.0",
    "os": "darwin 27.2.0",
    "arch": "arm64",
    "cpu": "Apple M5 Pro",
    "logicalCpus": 18,
    "colocatedNodes": 24,
    "replicasPerGroup": 3,
    "colocatedDrivers": 16,
    "colocatedControllers": 8,
    "colocatedNativeDrivers": 8
  },
  "options": {
    "offeredRate": 0,
    "groups": 8,
    "tenants": 2,
    "duration": 60,
    "warmup": 1,
    "drain": 60,
    "concurrency": 512,
    "workers": 4,
    "shops": 4,
    "hotShops": 1,
    "hotProbability": 0.8,
    "maxOrders": 96,
    "bakeMs": 250,
    "leaseMs": 2000,
    "duplicateRate": 0.1,
    "abandonRate": 0.1,
    "pollMs": 150,
    "requestTimeoutMs": 2000,
    "retryBudgetMs": 15000,
    "nodes": 3,
    "seed": "42",
    "initialization": "static",
    "queryRouting": "replicas",
    "readConsistency": "replica-local",
    "driver": "rust",
    "driverBinary": "/Users/pcarrier/src/flower/target/release/flower-bench-driver",
    "binary": "/Users/pcarrier/src/flower/target/release/flower",
    "json": "/Users/pcarrier/src/flower/bench/results/latest.json",
    "html": "/Users/pcarrier/src/flower/bench/results/latest.html",
    "baseline": null,
    "cpuProfile": null,
    "chaos": true,
    "http2": true,
    "keepData": false,
    "help": false
  }
}
Group failures and replica routing
[
  {
    "group": "group-0",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51182",
          "attempts": 153721,
          "completed": 152378,
          "failures": 1343
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51190",
          "attempts": 167854,
          "completed": 167854,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51194",
          "attempts": 167854,
          "completed": 167854,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 20215,
      "claims": 108,
      "emptyClaims": 858,
      "reclaimed": 12,
      "abandoned": 12,
      "lostLeases": 0,
      "staleLeaseChecks": 12
    }
  },
  {
    "group": "group-1",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51187",
          "attempts": 152151,
          "completed": 150889,
          "failures": 1262
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51202",
          "attempts": 166370,
          "completed": 166370,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51203",
          "attempts": 166370,
          "completed": 166370,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 19959,
      "claims": 104,
      "emptyClaims": 870,
      "reclaimed": 8,
      "abandoned": 8,
      "lostLeases": 0,
      "staleLeaseChecks": 8
    }
  },
  {
    "group": "group-2",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51184",
          "attempts": 153433,
          "completed": 152246,
          "failures": 1187
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51198",
          "attempts": 166432,
          "completed": 166432,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51199",
          "attempts": 166432,
          "completed": 166432,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 20265,
      "claims": 105,
      "emptyClaims": 856,
      "reclaimed": 9,
      "abandoned": 9,
      "lostLeases": 0,
      "staleLeaseChecks": 9
    }
  },
  {
    "group": "group-3",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51183",
          "attempts": 149898,
          "completed": 148144,
          "failures": 1754
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51191",
          "attempts": 166965,
          "completed": 166965,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51195",
          "attempts": 166965,
          "completed": 166965,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 20094,
      "claims": 109,
      "emptyClaims": 844,
      "reclaimed": 13,
      "abandoned": 13,
      "lostLeases": 0,
      "staleLeaseChecks": 13
    }
  },
  {
    "group": "group-4",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51181",
          "attempts": 151586,
          "completed": 150217,
          "failures": 1369
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51189",
          "attempts": 167240,
          "completed": 167240,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51193",
          "attempts": 167238,
          "completed": 167238,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 19958,
      "claims": 106,
      "emptyClaims": 838,
      "reclaimed": 10,
      "abandoned": 10,
      "lostLeases": 0,
      "staleLeaseChecks": 10
    }
  },
  {
    "group": "group-5",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51186",
          "attempts": 151714,
          "completed": 150491,
          "failures": 1223
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51196",
          "attempts": 167245,
          "completed": 167245,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51197",
          "attempts": 167244,
          "completed": 167244,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 19946,
      "claims": 107,
      "emptyClaims": 846,
      "reclaimed": 11,
      "abandoned": 11,
      "lostLeases": 0,
      "staleLeaseChecks": 11
    }
  },
  {
    "group": "group-6",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51180",
          "attempts": 151083,
          "completed": 149810,
          "failures": 1273
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51188",
          "attempts": 164659,
          "completed": 164659,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51192",
          "attempts": 164659,
          "completed": 164659,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 20072,
      "claims": 113,
      "emptyClaims": 833,
      "reclaimed": 17,
      "abandoned": 17,
      "lostLeases": 0,
      "staleLeaseChecks": 17
    }
  },
  {
    "group": "group-7",
    "failures": [],
    "queryRouting": {
      "mode": "replicas",
      "consistency": "replica-local",
      "auditConsistency": "fresh",
      "nodes": [
        {
          "id": 1,
          "url": "http://127.0.0.1:51185",
          "attempts": 151511,
          "completed": 150184,
          "failures": 1327
        },
        {
          "id": 2,
          "url": "http://127.0.0.1:51200",
          "attempts": 166930,
          "completed": 166930,
          "failures": 0
        },
        {
          "id": 3,
          "url": "http://127.0.0.1:51201",
          "attempts": 166928,
          "completed": 166928,
          "failures": 0
        }
      ]
    },
    "counters": {
      "issuedOrders": 96,
      "delivered": 96,
      "replayChecks": 20217,
      "claims": 110,
      "emptyClaims": 863,
      "reclaimed": 14,
      "abandoned": 14,
      "lostLeases": 0,
      "staleLeaseChecks": 14
    }
  }
]