Notes from the garden.

Plant your first database.

Start a local node, deploy a TypeScript application, and call a method. The server walkthrough runs from a checkout. TypeScript examples use the npm package: install @flower-js/sdk in your own project. In the checkout, npm run build builds its package exports. The complete SDK reference covers every public method, type, and tradeoff.

Build the server

git clone https://github.com/flower-js-org/runtime.git
cd runtime
npm ci
npm run build
cargo build --release --bin flower

Start one node

Choose an operator token and use the same value in both terminals. This local example binds to loopback and writes to its own data directory.

export FLOWER_ADMIN_TOKEN='local-development-secret'
./target/release/flower \
  --id 1 \
  --listen 127.0.0.1:7101 \
  --data .flower/node1

Initialize once, then deploy

The operator token protects bootstrap, deployment, and peer traffic. Ordinary method callers do not need this token.

export FLOWER_ADMIN_TOKEN='local-development-secret'
node sdk/cli.ts init --members 1=127.0.0.1:7101

# Check that the initial election has produced a leader.
curl -fsS -H "Authorization: Bearer $FLOWER_ADMIN_TOKEN" \
  http://127.0.0.1:7101/raft/metrics

node sdk/cli.ts deploy examples/orders.ts

Wait for state: "Leader" before deploying. An early UNAVAILABLE means the election is still settling; retry the operation. Initialization is only for a fresh cluster, never a restarted data directory.

Make a change. Read its result.

node sdk/cli.ts call order.create @examples/orders.create.json \
  --request-id create-order-42
node sdk/cli.ts call order.get '"order-42"'
# value: { order: { shippingCents: 500 }, subtotal: 3200, total: 3700 }

node sdk/cli.ts mutate order.updateLine @examples/orders.update.json \
  --request-id update-line-2
node sdk/cli.ts query order.get '"order-42"'
# value: { order: { shippingCents: 500 }, subtotal: 4600, total: 5100 }

node sdk/cli.ts watch order.get '"order-42"'

Run the homepage terrarium

After building, starting, and initializing the local node above, deploy the complete application from the homepage walkthrough, then run its client in another terminal:

node sdk/cli.ts deploy docs/terrarium.ts --initialization static
node docs/terrarium-client.ts

The client plants Luna in a shared moon garden and watches its materialized view. Each garden has twelve spots: the planting mutation checks its indexed population atomically, including concurrent gardeners. Durable callbacks bloom a seed after five seconds and remove it after thirty-five, making room for another planting. These deadlines mean “not before”; downtime can delay both transitions. Callbacks carry the planting time so an old timer cannot affect a replacement. The plant, both timers, and reactive updates commit together. Other viewers see changes through SSE; this query permits replica lag and watches may coalesce transitions. Keep the same request ID when retrying a plant; use a new request ID when replanting a freed spot. Stop the watch with Ctrl+C. Gardens in this demo are public, not authorization boundaries. Source: application and client.

Your application is the API.

Collections hold JSON. Derived values describe relationships. Methods decide exactly what callers may do.

Save this complete module as examples/counter.ts. The internal function names and public aliases deliberately differ.

import { collection, define, derive, mutation, query } from "@flower-js/sdk";

const counters = collection<number>("counters");

const doubled = derive("counter.doubled", (ctx, id: string) =>
  (ctx.get(counters, id) ?? 0) * 2,
);

const increment = mutation("internal.counter.increment", (ctx, id: string) => {
  if (typeof id !== "string" || !id) throw new TypeError("An ID is required");
  ctx.set(counters, id, (ctx.get(counters, id) ?? 0) + 1);
  ctx.materialize(doubled, id);
  return ctx.get(doubled, id); // Includes the write above.
});

const get = query("internal.counter.get", (ctx, id: string) => {
  if (typeof id !== "string" || !id) throw new TypeError("An ID is required");
  return {
    count: ctx.get(counters, id) ?? 0,
    doubled: ctx.get(doubled, id),
  };
});

export default define({
  definitions: [doubled],
  http: {
    "counter.increment": increment,
    "counter.get": get,
  },
});

node sdk/cli.ts deploy examples/counter.ts
node sdk/cli.ts call counter.increment '"visits"' --request-id visit-001
node sdk/cli.ts call counter.get '"visits"'
# value: { count: 1, doubled: 2 }

define({ collections, definitions, http, maintenance }) owns the complete method registry. Listing a method in http registers it automatically. Other definitions stay private. Replacing this bundle also replaces the public allowlist.

Reuse initialized application snapshots

For modules whose initialization does not depend on invocation bindings, add --initialization static to the build or TypeScript deploy command. Flower snapshots the initialized QuickJS heap and restores its logical state for every callback. New instances use Wasmtime copy-on-write images where supported; serial mutation batches may reuse exactly restored resident instances. Before capturing a reusable image, the host collects unreachable QuickJS cycles and restores its normal 50% allocation headroom. Automatic GC and configured memory/execution limits stay active. Frozen method and derived contexts are created during trusted setup, before application initialization; overriding Object.freeze in a module does not intercept their construction. Globals and closures still reset every time. Existing bundles keep per-invocation initialization and reuse the base sandbox plus cached bytecode.

A small context, ordinary TypeScript

get returns null for a missing record. scan returns {key, value} rows; equality queries return matching values. Both use lexical primary-key order. Types help authors, but application methods must validate their own arguments and record rules.

Follow the reads. Maintain the result.

Flower records the dependencies a function actually reads, then updates the affected graph with the source changes.

Materialize what should stay current

ctx.materialize(value, args) retains that derived instance and its transitive dependencies. Its identity is the definition name plus canonical JSON arguments. Omitted arguments mean null. Removing a root collects dependencies that no remaining root needs.

A query can read an unmaterialized derived value; Flower computes it temporarily without storing it. A mutation sees its own earlier writes, including through derived values. Source records, materialized outcomes, and the method result publish atomically.

When a changed derived child produces an identical outcome, its parent can reuse the previous outcome. Branch dependencies still update, with normal cycle checks and collection. Multiple potentially changed branches execute in application order to preserve dynamic control flow; invalidation discovery still visits their dependency graph.

Choose useful dependency boundaries

A missing record is still a dependency. Scans depend on the whole collection. Declare indexed collections in define({ collections: [orders], ... }) to persist equality indexes: readers then depend on their matching bucket and rows, including inserts into an empty bucket. Undeclared collections retain temporary, scan-based indexes. Split expensive order statistics from shop accounting, for example, so a tip can reuse unchanged order totals.

Maintain totals from row deltas

aggregate defines an indexed group accumulator with pure initial, add, and remove callbacks. Rust applies only changed rows to retained totals, and replicates index entries and results atomically. Materialize and read it like any derived value. Use deterministic inverse operations, such as integer sums and counts. Deployment prepares indexes and aggregate rebuilds off the writer lane by default, then atomically publishes only if its base revision is unchanged. A concurrent write returns DEPLOYMENT_CONFLICT; retry the same request ID or use client.deploy(bundle, {requestId, preparation: "blocking"}) for an exclusive preparation window. For a large collection, use the resumable staged flow below. Direct preparation and materialized-root rebuilding still obey normal execution and transaction budgets. The indexes and reducers guide has the complete example and operating details.

Change computation code, then activate it

Editing a local derive or aggregate callback has no effect on a running database until you build and deploy its bundle. A direct deployment recomputes retained materializations in one complete candidate and publishes the code and results atomically; it does not wait for each query to refresh them. For a larger graph, staged deployment prepares durable pages while the current application keeps serving, then switches to the prepared graph on explicit activation.

1. Stage and backfill. stageDeployment(bundle, {requestId}) records the target bundle and added index definitions. advance backfills bounded source pages while ordinary writes maintain active and added indexes atomically. A code-only change starts in rebuilding.
2. Rebuild. Each advance prepares an adaptive batch of retained roots and their required dependencies in a hidden graph generation. Source rows are stored once. The active mutation method runs once; its final source/root changes also drive target derived callbacks. Active and already-built target results commit together. Public requests use the active code, policy and graph.
3. Ready. The target graph has finished its root traversal. Subsequent mutations keep it current synchronously, including added or removed roots; they do not send the job back to rebuilding or create a catch-up queue. This extra maintenance continues until activation, cancellation or a fatal target-maintenance failure.
4. Activate. activate requires ready and atomically switches the bundle, public aliases, policy, key declarations, indexes and graph pointer. It reuses prepared clock-independent outcomes and refreshes clock/key-dependent work within normal budgets.
5. Collect. Repeat bounded collect calls until collected to reclaim obsolete or canceled-build graphs and indexes. Active generations and shared index definitions are preserved. Finish this cleanup before another deployment.

Use a unique request ID for each deployment intent and save it with its bundle. Status, index cursor, graphCursor and rebuiltRoots survive leader changes, complete restarts, snapshots and logical-partition moves. Verify both the job ID and bundle hash before resuming rather than starting another. The counters describe work completed, not a completion percentage. There is no automatic page worker or activation.

const requestId = "application-upgrade-v2"; // Save and reuse for this bundle.
const maxBytes = 256 * 1024;
let build = (await client.stagedDeploymentStatus()).value;
if (build === null || build.phase === "collected") {
  build = (await client.stageDeployment(bundle, {requestId})).value;
}
if (build.requestId !== requestId || build.bundleHash !== bundle.hash) {
  throw new Error("Existing deployment ID/bundle does not match this intent");
}
while (build.phase === "backfill" || build.phase === "rebuilding") {
  build = (await client.controlStagedDeployment({
    operation: "advance", requestId, maxBytes,
  })).value;
}
if (build.phase === "failed") {
  throw new Error(build.error ?? "Cancel and collect the failed target");
}
if (build.phase === "ready") {
  build = (await client.controlStagedDeployment({
    operation: "activate", requestId,
  })).value;
}
while (build.phase === "active" || build.phase === "canceled") {
  build = (await client.controlStagedDeployment({
    operation: "collect", requestId, maxBytes,
  })).value;
}

After an uncertain response, read status and continue with the same deployment ID. Retrying advance may do the next page; it does not replay an identical page result. Aborting an HTTP request does not undo committed progress. A canceled intent stays canceled and cannot activate; use a new request ID for corrected code after collection. Once activated, roll forward with another deployment.

An explicit page or activation error leaves earlier committed progress intact and does not switch the active application. Adjust the relevant budget and retry when appropriate, or cancel and collect before staging corrected code. A fatal error during target maintenance inside an ordinary mutation instead records phase: "failed" and error; the active mutation can still commit if its result and the failure status fit the normal budgets. A failed job cannot activate: cancel and collect it. Ordinary derived exceptions retain their stored-error behavior. While a cross-group transaction is prepared, state-changing deployment controls return TRANSACTION_PREPARED; status remains inspectable.

Choose rebuild throughput and write latency independently

Each graph page occupies the logical writer during preparation and commit; reads can continue from committed snapshots. Larger pages reduce commit overhead but can cause longer competing write stalls. FLOWER_DEPLOYMENT_PAGE_MS independently sets the soft graph-page preparation target: positive integer milliseconds, default 200 ms, capped by FLOWER_EVALUATION_TIMEOUT_MS. Lower values target more, smaller pages and shorter stalls; larger values favor fewer pages and rebuild throughput. Start with 20 ms for a latency-sensitive experiment and measure the application workload.

export FLOWER_DEPLOYMENT_PAGE_MS=20
export FLOWER_WRITER_BATCH_MS=200
# Start or restart each Flower node with these settings.

These settings are read at startup; restart nodes to apply changes. The deployment setting leaves ordinary writer batching unchanged. maxBytes is a separate request-level bound on inspected index work or the graph patch; it defaults to the configured transaction byte budget. Exact command size, output and admission limits also apply. Graph pages begin with one root and adapt from observed time and bytes, growing by at most twice per page.

The page target is not an end-to-end deadline. An unsuccessful multi-root candidate may retry only its first root using the full normal evaluation timeout, so a page can take its target plus that evaluation allowance, admission and commit time. One root and its newly reached dependency closure must still fit one evaluation; callbacks and initial aggregate-group scans are not resumable. The full retained graph metadata must fit memory. Append-only growth reuses a cached depth proof, while removals, changes to existing derived edges and restored metadata can still require whole-graph validation. Activation clock/key refresh remains bounded by normal evaluation limits.

Completed target roots add derived maintenance work until activation, cancellation or a fatal target failure; the extra graph consumes storage until collection. Speculative mutation preparation is bypassed while a target graph is maintained, while serial batches and grouped commits remain available. Pure rebuild-progress updates do not invalidate query caches. The staged benchmark methodology and historical measurements show the throughput/latency tradeoff, sample counts and limitations. Those debug, warm-code, single-host measurements used the earlier shared writer setting; they are not measurements of this dedicated setting or production capacity.

Migrate stored records explicitly

Deploying code rebuilds derived state; it does not automatically transform source records. Staged deployment stores source rows once, so both active and target computations must understand any intermediate record shapes. The target mutation method body is not replayed: Flower applies the active method's final source/root changes to target derived callbacks. If compatibility requires writing two source representations, implement that explicitly in the application.

1. Deploy compatibility code. Read old and new record versions, make ordinary writers produce the new version, and expose an application mutation that performs a bounded migration batch.
2. Commit progress with the data. In that mutation, read a durable migration cursor/version, transform a bounded page, and store the next cursor with the changed rows. Both the transformation and its progress commit atomically. Preserve the batch request ID on an uncertain retry, and guard by record version so resumed work does not convert a row twice.
3. Use stable traversal. For every record in a collection, page by source key with ctx.scan(rows, {gt: afterKey, limit: 100}), omitting gt on the first page. Read afterKey from the durable migration record inside the mutation and save the last returned key with the converted rows; an empty page can mark traversal complete. Ordinary writers must already produce the new shape so inserts behind the cursor are safe. For a selected indexed subset, use a declared index and ctx.range ordered by fields the migration does not change. Range cursors use each invocation's current snapshot; moved rows can be skipped or repeated, and missing or nonscalar indexed fields are excluded. Avoid growing offsets and repeated unbounded scans.
4. Verify, then remove compatibility. Check remaining old versions and application invariants independently. Stage and activate the final code only after every required row is compatible, then remove obsolete source fields or representations through explicit bounded mutations.

Names are durable identities: editing a TypeScript interface or changing a collection/derived name does not convert or rename existing rows or retained roots. Use explicit source copies and a compatibility transition; materialize new roots and unmaterialize old ones through application mutations before removing old definitions. Protect migration methods with application authorization; an operator token alone does not make a public mutation admin-only.

The migration's cursor and version are application records, distinct from a staged deployment's graphCursor. Its batch size must account for reactive work as well as row bytes; while a target graph is present, each migration mutation also maintains its completed roots. There is no built-in arbitrary migration callback or automatic catch-up for source conversion. Moving a logical partition between Raft groups preserves its stored data and code; it is separate from changing record shapes. See the range/cursor contract, mutation context and staged deployment API.

Errors have defined behavior

An ordinary derived exception becomes an error outcome and propagates to readers; a method may catch it. An uncaught method exception discards the whole mutation. Cycles and shared transaction-budget failures abort even if application code catches an exception. Ordinary JavaScript errors can still be caught; an exhausted transaction budget cannot. Returning an error-shaped object is still a successful result.

Call methods. Keep your receipts.

Use the TypeScript client, CLI, or a small JSON HTTP request. The deployed code chooses the behavior.

Save a client script at the checkout root. Use any reachable member as the primary address; servers forward writes to the current leader. Optional read endpoints let queries and watches run across replicas.

import { FlowerClient } from "@flower-js/sdk";

const client = new FlowerClient("http://127.0.0.1:7101");
const result = await client.call("counter.increment", "visits", {
  requestId: "visit-002",
});
// { revision, value, duplicate }

console.log(result);
console.log(await client.call("counter.get", "visits"));

curl -fsS http://127.0.0.1:7101/v1/call \
  -H 'Content-Type: application/json' \
  -d '{"name":"counter.increment","args":"visits","requestId":"visit-003"}'

All three return {revision, value, duplicate}. Mutations require a request ID over HTTP; the SDK supplies one when omitted. Queries return duplicate: false. A query through /v1/call ignores a supplied request ID and rejects expectedRevision.

Check the caller before looking up the receipt

Register a private query as define({authorize}). Rust runs it before public methods, cached results, receipt replay and watch refresh. Keep credentials outside business arguments, so refreshing a token preserves the same retry identity.

import { define, query, key, jwt } from "@flower-js/sdk";
const sessions = key("sessions", { algorithm: "Ed25519", usages: ["verify"] });
const authorize = query("admission", (_ctx, request) => {
  const { claims } = jwt.verify(request.credentials, sessions);
  if (typeof claims.sub !== "string" || typeof claims.tenant !== "string") return null;
  return { subject: claims.sub, tenant: claims.tenant };
});
export default define({ keys: [sessions], authorize, http: { /* business methods */ } });

Methods read ctx.principal(); callers cannot supply that principal. A named partition requires its name to match the admitted tenant. Return null/throw to deny. A hook can read current policy, but state-changing checks stay in the mutation. Protected reads currently require fresh policy even when their data method allows replica-local reads. Watches close when refresh detects expiry/revocation. See delegation, client credentials, and the complete contract.

Keep replay fences after forgetting results

Operator-controlled retry epochs let Flower bound retained results without executing old requests again. Initialize a history through controlRetention, use a client with boundedRetries: true, and persist await client.newRequestId() before sending an uncertain business operation. Advance the minimum epoch and run bounded collection only after its promised retry window has ended.

Epochs are explicit replicated windows, not automatic clock TTLs. An expired retry returns RETRY_WINDOW_EXPIRED: its mutation may have committed. Never replace that ID silently; resolve uncertainty through a business identifier. New work can explicitly refresh the client's epoch. Receipt GC preserves transaction decisions and does not shrink every physical file or backup. The retention API describes byte budgets, migration and restore restrictions.

Long-running clients can openRetrySession, persist sequence allocation, and acknowledgeRetrySession only after durably consuming a contiguous result prefix. Acknowledged sequences never execute again. workQueue claims include a history identity when retention is initialized; forward it unchanged on completion/failure. After disaster restore, a new incarnation rejects old claims even before their old deadline. External sinks must also enforce the accepted history and increasing fencing token; lease expiry alone cannot stop a paused worker.

One transaction, several groups

transaction(name, args => ({ calls: [...] })) exposes a pure plan of methods in named logical partitions or configured groups’ root databases. Participants stage their changes privately; a replicated commit decision makes them durable, and the coordinator keeps a retry receipt. A prepared logical database blocks fresh reads and writes until it resolves the decision; unrelated named partitions on the same physical group remain available. Opt-in replica-local reads can still observe older state. The transaction guide covers configuration, a transfer example, failure recovery, and this availability tradeoff.

Spread reads, keep them fresh

Queries and watches can execute on any replica. By default they are linearizable: the serving node obtains a quorum-backed read fence from the leader and waits for its own committed state to apply that fence before capturing a snapshot. Query CPU can run across the cluster; fresh reads still need a working quorum.

import { FlowerClient } from "@flower-js/sdk";

const client = new FlowerClient("http://127.0.0.1:7101", {
  queryUrls: [
    "http://127.0.0.1:7101",
    "http://127.0.0.1:7102",
    "http://127.0.0.1:7103",
  ],
});
const [first, second] = await Promise.all([
  client.query("counter.get", "visits"),
  client.query("counter.get", "clicks"),
]);
const updates = client.watch("counter.get", "visits");

queryUrls selects endpoints round-robin for queries and new watch, watchDeltas, and watchPoll subscriptions. Each watch stays on its chosen endpoint. Mutations, deployments, and generic call() use the primary URL, which may be any reachable member. The server forwards writes to the current leader without changing request IDs; no SDK leader selection is needed. Other operator APIs may have their own leader requirements. Query calls sent directly to /v1/call work on followers, but use client.query() to distribute SDK reads. There is no automatic retry, member discovery, or change in consistency. The HTTP/2 transport pools one session per origin.

Choose replica-local reads in application code

When a query may tolerate lag, declare that policy on the query itself and expose it normally:

const getLocal = query("internal.counter.local", (ctx, id: string) =>
  ctx.get(counters, id) ?? 0,
  { consistency: "replica-local" },
);
// Include "counter.local": getLocal in define({ http: ... }).

Replica-local queries and watches read one coherent, locally applied committed snapshot without a per-read quorum fence. After startup recovery, they can remain available during a partition, with no bounded-staleness guarantee. If a restarted node’s durable log extends beyond its recovered application checkpoint, it first needs a quorum-confirmed fence and local replay before serving these reads. Data, code, and the HTTP registry may all lag; removing an alias takes effect on a disconnected replica only after it applies that deployment. Calls on different replicas can return decreasing revisions. Omitting the option, or choosing linearizable, preserves fresh reads. HTTP callers cannot override the policy, and mutations and maintenance cannot declare it.

Use HTTP/2 from Node.js

The same port accepts HTTP/1.1 and cleartext HTTP/2 with prior knowledge (h2c). Configuring FLOWER_TLS_CERT_FILE, FLOWER_TLS_KEY_FILE and FLOWER_TLS_CA_FILE switches the listener to native TLS with h2/HTTP1 ALPN. Internal clients then require HTTPS and verify the configured CA and hostname. HTTP/1.1 Upgrade is not supported.

import { FlowerClient } from "@flower-js/sdk";
import { createHttp2Transport } from "@flower-js/sdk/http2";

const transport = createHttp2Transport({ requestTimeoutMs: 10_000 });
const client = new FlowerClient("http://127.0.0.1:7101", {
  fetch: transport.fetch,
});
try {
  console.log(await client.call("counter.increment", "visits", {
    requestId: "visit-over-h2-001",
    signal: AbortSignal.timeout(5_000),
  }));
} finally {
  await transport.close();
}

The package import is @flower-js/sdk/http2. This Node-only adapter accepts http:// and https:// origins, pools a multiplexed session per origin, and buffers bounded JSON responses. Ordinary JSON requests have a 30-second default whole-request timeout. SSE watches expose a streaming body; that timeout covers headers only, then the caller’s signal or transport closure controls lifetime. The adapter exposes maxRequestBytes (8 MiB default), maxResponseBytes (64 MiB), maxSessions (16), requestTimeoutMs, and idleTimeoutMs (30 seconds each). Byte allowances must fit Node’s buffer representation; timer values must fit its signed 32-bit millisecond timer. Streaming SSE uses the watch options for event/value limits. Close the transport when finished; closing cancels active streams. It does not retry, redirect, or downgrade automatically.

Cancellation stops waiting; it cannot undo a mutation that already committed. Reuse the same request ID after an uncertain result. For HTTPS, optional ca accepts PEM strings, Uint8Arrays or arrays of either; omitted roots use Node’s trust configuration. Certificate and hostname verification cannot be disabled.

curl --http2-prior-knowledge -i http://127.0.0.1:7101/health

Retry one operation with one identity

After a timeout, connection loss, election, or 503 UNAVAILABLE, retry the same request ID and content against any reachable member. A committed retry returns its original value and revision with duplicate: true. Preserve arguments, alias, and any expected revision. Do not replace an uncertain write with a new ID.

Use expectedRevision for a compare-and-set against the application’s current revision. A new intended operation gets a new ID. Separate CLI commands generate new IDs unless you pass --request-id.

Watch the value, stream the changes

POST /v1/watch accepts {name, args?} for an exposed query. The first SSE event is a full snapshot. Later events carry RFC 6902 JSON Patch deltas, or another snapshot when replacement is cheaper. The SDK’s watch reconstructs the value for you.

const stop = new AbortController();
for await (const { revision, value } of client.watch("counter.get", "visits", {
  signal: stop.signal,
})) {
  console.log(revision, value);
  // Break the loop or call stop.abort() to disconnect.
}

watchDeltas exposes the wire events directly: snapshots contain sequence, revision, value; patches contain sequence, baseSequence, revision, patch. The SDK adds a type field to identify them. Sequence numbers belong to the shared producer: an initial snapshot can start above zero, and a full reset can skip numbers. Patches must be consecutive and name the exact preceding baseSequence. Unchanged values emit nothing even when another mutation advances the revision; time-dependent values can change at the same revision.

Each subscriber checks the query allowlist, declared consistency, and its own authorization on refresh and idle ticks. Identical invocations share evaluation, diffing, and immutable encoded updates only within the same logical database, full admitted principal, deployment, and consistency. Credentials are never shared. A lagging subscriber gets a full reset snapshot if it missed a delta; deployment or principal-scope changes end the stream with WATCH_SCOPE_CHANGED. Fresh watches obtain a quorum-backed fence for every evaluation and check quorum on an idle tick (250 ms by default, FLOWER_WATCH_REFRESH_MS); replica-local watches use local state and the local registry. Commits wake watchers, and the tick catches clock changes. Load can delay the tick. There is no fixed subscription-count cap. Each stream retains at most one queued update plus its terminal event. Updates send immediately while the queue has room. When a changed update finds the queue full, the subscriber waits for space and batches intermediate changes into one refresh of the latest value, reacquiring admission, its declared read fence and current authorization before sending. Consumers waiting for room to send that update close after the configured send timeout (five seconds by default). Coalescing can skip intermediate commits.

Abort or break the loop to close the watch. Terminal server errors become FlowerError. Reconnect explicitly after a terminal error or transport loss, using any reachable replica that can serve the declared policy. Each new watch obtains its own read fence and begins with a full snapshot at the producer’s current sequence, with no retained history or Last-Event-ID replay. Configured read endpoints spread new subscriptions; an existing stream never moves between nodes. Use watchPoll(query, args, {intervalMs, signal}) when you explicitly want polling. Its interval is an integer from 1 to 2,147,483,647 milliseconds, the runtime timer range; larger values are rejected instead of overflowing into a busy loop. HTTP/1.1 and the Node HTTP/2 adapter both support streaming watches.

Client allowances are local options on both watch and watchDeltas: maxValueBytes defaults to 16 MiB of serialized UTF-8 JSON, maxEventBytes to 17 MiB per SSE event or HTTP error body, and maxPatchOperations to 256. Reconstruction checks the complete resulting value, not only the delta. Set positive safe integers when your server permits larger results; these options do not change server policy and are never sent over HTTP. For example, use {maxValueBytes: 32 * 1024 ** 2, maxEventBytes: 33 * 1024 ** 2} for a 32 MiB client allowance. The CLI accepts the same local policy through watch --max-value-bytes 33554432 --max-event-bytes 34603008 --max-patch-operations 512; these flags apply only to watch.

Fresh queries establish a quorum fence before capturing a snapshot, including on cache hits. Replica-local queries use their local applied snapshot. Successful clock-independent results may survive unrelated revisions: certificates validate the records, index membership, and derived outcomes actually read against that snapshot, plus code/schema/policy. Missing keys and empty ranges are tracked too. Identical concurrent reads can share their first evaluation. Every request checks its snapshot’s HTTP allowlist and declared policy and runs authorization before cache lookup; full principals separate entries. Retention is bounded by FLOWER_QUERY_CACHE_BYTES (16 MiB per logical database by default), and active shared-evaluation metadata by FLOWER_QUERY_FLIGHT_BYTES (512 KiB); zero disables either. Certificates are process-local and do not weaken snapshot consistency or Raft durability. Direct or transitive ctx.now() reads disable result reuse.

Public HTTP cache hits can bypass heavy preparation through a bounded probe. FLOWER_QUERY_WORKERS bounds these probes and authorization callbacks; heavy query execution shares FLOWER_PREPARATION_WORKERS with writers and watches. Probes try a slot immediately and reserve input/key bytes. A miss or busy probe releases its snapshot before ordinary fair admission captures a new one. Applications with authorization hooks or managed keys use the full admitted path. Encoded hit responses remain byte-accounted through transport, even after cache eviction, and retain no snapshot or evaluation slot. SSE admission is unchanged.

Let business logic bloom later.

Schedule an ordinary TypeScript mutation after an update. Its deadline and the business change commit together.

This complete example marks an invoice ready five seconds after its last update. Save it as examples/invoices.ts, then deploy it with the same CLI. This is a separate application bundle.

import { collection, define, mutation, query } from "@flower-js/sdk";
import { scheduler } from "@flower-js/sdk/scheduler";

const invoices = collection<{ total: number; status: string }>("invoices");
const finalize = mutation("internal.invoice.finalize", (ctx, id: string) => {
  const invoice = ctx.get(invoices, id);
  if (invoice) ctx.set(invoices, id, { ...invoice, status: "ready" });
  return null;
});
const timers = scheduler("invoiceTimers", { finalize });

const update = mutation("internal.invoice.update", (ctx, args: { id: string; total: number }) => {
  if (!args || typeof args.id !== "string" || !args.id ||
      !Number.isSafeInteger(args.total) || args.total < 0) {
    throw new TypeError("An invoice ID and nonnegative integer total are required");
  }
  ctx.set(invoices, args.id, { total: args.total, status: "draft" });
  return timers.after(ctx, `finalize:${args.id}`, 5_000, "finalize", args.id);
});
const get = query("internal.invoice.get", (ctx, id: string) => ctx.get(invoices, id));

export default define({
  collections: [timers.records],
  maintenance: timers.maintenance,
  http: { "invoice.update": update, "invoice.get": get },
});

node sdk/cli.ts deploy examples/invoices.ts
node sdk/cli.ts call invoice.update '{"id":"inv-1","total":2400}' \
  --request-id invoice-edit-1
node sdk/cli.ts watch invoice.get '"inv-1"'

Declare timers.records in the application collections so maintenance seeks the first pending deadline instead of scanning history. Reusing the timer ID replaces its pending deadline. Use a new ID for each update when every update should trigger its own callback. timers.at accepts an absolute epoch-millisecond deadline; cancel, get, scan, and retry manage pending or failed timers.

Timers survive. Closures do not travel.

Timer records contain a handler alias and JSON arguments. The current deployed code resolves that alias; captured closures are not serialized. Keep an old handler alias or migrate pending records when renaming it. A callback can schedule its next occurrence under the same ID.

The leader polls maintenance every 250 ms by default (FLOWER_MAINTENANCE_INTERVAL_MS) and can process bounded catch-up bursts. Each callback keeps its own revision and rollback boundary; successful patches can share a durable Raft commit. A failed callback discards its changes, then a separate private TypeScript handler records failure and applies retry policy. Defaults: three attempts, 1,000 ms initial backoff, 60,000 ms cap.

The existing examples/scheduling.ts adds publication status, cancellation, and retry methods. General maintenance can use define({ maintenance }) and return { $flower: { continue: true } } to request another invocation. Catch-up stops on an idle patch, the configured transaction byte budget, or an invocation that takes the burst past FLOWER_MAINTENANCE_BURST_MS (50 ms by default). There is no independent invocation-count ceiling.

Give work a lease. Give keys a lifetime.

Both policies are TypeScript helpers built on ordinary records, methods, and the same transactional clock.

Claim → work → complete or fail

import { expiringCollection, workQueue } from "@flower-js/sdk/temporal";

const jobs = workQueue("jobs", { maxLeaseMs: 30_000 });
const sessions = expiringCollection("sessions", {
  expiration: { afterUpdateMs: 60_000 },
});
const tokens = expiringCollection("tokens", {
  expiration: { afterCreationMs: 10_000 },
});

Include each helper’s records once in define({collections: [jobs.records, sessions.records, tokens.records]}) to maintain its durable indexes. Timer selection, expiry sweeps, and lease selection then use bounded ordered pages. Omitted declarations retain correct Rust scan fallback. Queue scope separates tenants within one collection; use workQueue("jobs", {maxLeaseMs: 30_000, scope: tenant}), with composite backing keys and queue.scan(ctx) for scoped inspection. Claims preserve creation-order FIFO and inspect expired leases; large expiry herds still consume transaction work. Scopes do not authenticate tenants.

Expose mutation methods around enqueue, claim, complete, fail, and retry. The supplied examples/workers.ts already does this; deploy it to a fresh application database.

node sdk/cli.ts deploy examples/workers.ts
node sdk/cli.ts call jobs.enqueue \
  '{"id":"receipt-1","payload":{"invoice":"inv-1"}}' \
  --request-id enqueue-receipt-1

import { FlowerClient } from "@flower-js/sdk";
import type { Claim } from "@flower-js/sdk/temporal";

const client = new FlowerClient("http://127.0.0.1:7101");
const args = { owner: "worker-1", leaseMs: 10_000 };
const claimId = crypto.randomUUID(); // Persist this for retries of this call.
const { value: lease } = await client.call<typeof args, Claim | null>(
  "jobs.claim", args, { requestId: claimId },
);

if (lease) {
  // Perform external work here, using idempotency or downstream fencing.
  // Preserve this completion ID and its exact arguments on uncertain replies.
  const completionId = crypto.randomUUID();
  await client.call("jobs.complete", {
    id: lease.id, owner: lease.owner, token: lease.token,
    result: { processed: true },
  }, { requestId: completionId });
}

A claim returns {id, payload, owner, token, expiresAt, attempt} or null. Completion and failure must present the current owner and fencing token before its deadline. Expired work is claimable again without waiting for maintenance; an old worker fails with a LEASE_LOST application error. Failed jobs remain inspectable until explicitly retried.

A repeated claim request returns its original receipt, which may already be expired. Use a new request ID to acquire fresh work. Each claim has a configured maximum duration; renewal is not implemented. Expiry cannot stop a worker process or undo an external effect, so the receiving service should accept idempotency keys or enforce fencing tokens.

Expiry is a read rule, cleanup is a later action

Expiring collections hide a key when ctx.now() >= expiresAt. entry exposes value and creation/update/expiry timestamps. sweep removes expired storage when invoked by a mutation or private maintenance.

Updating a live record preserves its creation timestamp; replacing an expired one starts a new lifetime. To refresh expiry on access, expose a mutation that reads and rewrites it. A query never silently extends a lifetime. Use a scheduled callback when expiration should run other business logic as well as deletion.

ctx.now() is fixed throughout one evaluation. Nodes must keep their clocks synchronized. Query timestamps may move backwards between serving nodes; fresh-read guarantees concern state, not synchronized clocks. Lease and expiry checks use invocation time, so execution and commitment can consume part of the duration before the response arrives.

A seal without a secret in the petals.

Declare what your code needs. Let an operator grant it. Rust owns private material, version policy and reusable crypto contexts; QuickJS sees public handles and operation results.

import { define, key, mutation, query } from "@flower-js/sdk";
import { jwt } from "@flower-js/sdk/crypto";

const sessions = key("sessions", {
  algorithm: "Ed25519", usages: ["sign", "verify"],
});
const issue = mutation("session.issue", (ctx, user: string) => {
  // Authenticate the caller and authorize this user before issuing a token.
  if (typeof user !== "string" || !user) throw new TypeError("user required");
  return jwt.sign({ sub: user, iss: "shop", aud: "shop-api",
    exp: ctx.now() / 1000 + 900 }, sessions);
});
const check = query("session.check", (_ctx, token: string) =>
  jwt.verify(token, sessions, { issuer: "shop", audience: ["shop-api"] }).claims);

export default define({ keys: [sessions], http: { issue, check } });

The example illustrates issuance, not caller authentication: add your application’s authorization before exposing issue. The frozen sessions descriptor contains only a name, algorithm and permitted uses. A deployed declaration and a separate operator binding must both authorize an operation. JWT signing infers EdDSA from this key; verification requires a valid managed version kid and still checks expiration, issuer and audience against the invocation clock.

# Install the same protected wrapping file on every authorized server.
umask 077
openssl rand 32 > /secure/flower-wrapping.key
# Start each server with FLOWER_KEYRING_FILE=/secure/flower-wrapping.key.

# SDK CLI: supply FLOWER_URL and FLOWER_ADMIN_TOKEN for your cluster.
flower key generate session-signing --algorithm Ed25519 --request-id create-session-key
flower key bind sessions session-signing --usages sign,verify --request-id bind-sessions
flower deploy sessions.ts --request-id deploy-sessions
flower key rotate session-signing --request-id rotate-sessions-2
flower key list
flower key cache

# Native runtime executable: plaintext stays on this local machine.
/path/to/native/flower key seal --wrapping-key-file /secure/flower-wrapping.key \
  --format pem < private.pem > sealed.json
# SDK CLI sends only the authenticated encrypted envelope.
flower key import imported-signing sealed.json --algorithm Ed25519 \
  --request-id import-signing

For a named tenant database, add --partition north to SDK deploy/key commands, or use client.partition("north"). The native seal command runs locally and only writes an encrypted envelope; SDK import accepts that JSON, never a raw secret. The default listener is h2c; enable native TLS to protect operator credentials and metadata. The wrapping file is a separate 32-byte secret: install it only on authorized nodes and back it up independently.

Use the same API. Keep the private bytes native.

Managed Ed25519 keys support JWT and NaCl signing/verification. P256, RSA and HS256 support JWT; A256GCM supports encrypted JWT; XSalsa20Poly1305 and X25519 support applicable NaCl operations. publicKey(handle) requires an explicit publicKey grant. nacl.box.before(peerPublic, privateHandle) returns an opaque SharedKey usable only within that callback, with derive plus encrypt/decrypt grants. It has no observable random token, cannot be serialized or forged, and cannot be converted into private bytes. For raw NaCl ciphertext, save keyVersion(key).version alongside it and select that version on decryption after rotation. Retire preserves verification/decryption; revoke blocks use; keyDestroy removes current encrypted material while leaving a tombstone and does not erase backups. Raw Uint8Array overloads remain available for interoperability.

Native randomness and automatic JWT nonces require a mutation; explicit unique nonces make pure operations usable in queries. NaCl nonces are 24 bytes; AES-GCM nonces are 12. nacl.setPRNG controls only the guest-local NaCl override, never operator key generation or automatic JWT nonces. Because each callback starts with a fresh guest, unsafe mutable generators can repeat outputs. Flower uses host OS entropy for fresh material, not a seeded consensus RNG or persistent QuickJS PRNG state. The leader commits resulting changes. Explicit-input signatures are deterministic (P256 uses RFC 6979); RSA blinding stays native. Native shared handles expose no random identity and cannot become persistent capabilities.

Rotate the key. Keep the policy fresh.

Rotation selects a new immutable version for signing/encryption. Managed JWT/JWE carries that version in its authenticated kid, so old tokens can verify/decrypt while their version remains permitted and unrevoked. NaCl has no kid; persist keyVersion(key).version alongside ciphertext and use keyVersion(key, savedVersion) for historical verification/decryption. That handle cannot sign, encrypt or derive a new shared encryption capability. flower key revoke NAME --version N disables one version; omit --version to revoke every existing version. Revoking the active version requires a later rotation before new operations can use it.

All query/watch evaluations become quorum-backed when the application declares managed keys, even aliases marked replica-local. Each callback pins its admitted policy; revocation cannot retroactively cancel already admitted work. Managed dependencies invalidate query results and recompute materialized derived values atomically with policy changes. Locked or revoked dependencies fail rather than returning an obsolete secret-dependent result.

Raft replicates changes: encrypted key catalog metadata, derived results and retry receipts. Generation and encryption happen before proposal; followers apply the committed changes without rerunning TypeScript or generating another key. Private bytes do not enter ordinary collections, bundles, responses or the log.

Raft log compaction and snapshots avoid keeping every historical mutation forever; current records are replaced/deleted normally. Without retention initialization, retry receipts retain fingerprints and results indefinitely. Opt-in epochs and session ACKs reclaim results while preserving replay fences. Key rotation likewise retains encrypted old versions, and revocation does not delete them. Version garbage collection is explicit; keyDestroy removes current encrypted material, and keyRewrap rotates its wrapping key without changing key/version identity. These histories remain in snapshots and require capacity planning.

Reuse the expensive parts.

The node caches actual native signers, verifiers, HMAC/AES contexts and zeroizing key buffers across isolated callbacks. FLOWER_KEY_CACHE_BYTES defaults to 16 MiB; zero disables retention. FLOWER_KEY_CACHE_TTL_MS defaults to 0 (no expiry); a positive value reloads expired contexts on use. The first resolution checks the callback’s pinned policy; repeated uses reuse its authorized context and check the pinned permissions. There is no fresh Raft barrier for each crypto function. The TTL bounds prepared-context reuse age, including this callback-local fast path; it does not guarantee erasure from memory at a deadline. Claims validation and plaintext results are never cached here. flower key cache reports the ingress node’s global cache, including under a partition URL. Its hit counts exclude callback-local reuse that bypasses that node cache.

The wrapping file is loaded at startup. A cache TTL is not an external revocation lease, and changing the file requires restart. Migration moves encrypted envelopes with the partition; a destination must unlock all unrevoked versions before cutover. Missing or wrong wrapping keys pause the move until corrected. For KEK rotation, mount the new FLOWER_KEYRING_FILE plus previous protected files in FLOWER_KEYRING_PREVIOUS_FILES (a JSON array), restart, call keyRewrap on every catalog, then remove old providers and restart. KMS/HSM backends and general exportable secrets remain unimplemented.

Private material stays out of application JS, but an authorized host still holds it in memory. Existing byte, native-memory and evaluation budgets apply; native calls check deadlines before and after execution rather than being interrupted halfway. The complete managed-key reference covers every method, import format, permission and operating limit; the raw NaCl/JWT reference covers all functions and constants.

Operate a small, real Raft cluster.

The leader evaluates mutations and a quorum commits their changes. Followers apply those changes without rerunning mutations. Every replica can execute query methods.

One node-wide preparation pool serves heavy work across all logical partitions. Trusted partition lanes rotate fairly; query evaluations and watch refreshes acquire capacity before capturing their execution snapshot. The public HTTP cache probe uses separate bounded lookup slots and the same retained-byte budget; it is best-effort rather than a separate fair partition scheduler. FLOWER_PREPARATION_WORKERS, FLOWER_PREPARATION_MEMORY_BYTES and FLOWER_QUEUED_INPUT_BYTES bound admitted work and retained request/watch values; operator and maintenance work have separate reserves. A query or watch waiting for an identical producer retains only request bytes, then reacquires a fresh snapshot and authorization before resuming. A slow fresh-read barrier occupies its preparation reservation or cache-probe slot, so more concurrency trades memory for latency rather than creating capacity. Slow cache-hit response consumers retain output bytes until transport releases them. Adaptive batches flush earlier as requests age and maintenance becomes due, with a default 200 ms maximum preparation window. Inspect GET /admin/resources with the operator token and see the complete settings reference.

Start three distinct nodes

Use a fresh cluster for this walkthrough. Stop the earlier single-node server with Ctrl+C first so port 7101 is available. Run each command in its own terminal, with the same operator token in all three. A node’s ID, address, and exclusive data directory must be distinct.

export FLOWER_ADMIN_TOKEN='replace-with-a-shared-operator-secret'
# Run one of these commands per terminal:
./target/release/flower --id 1 --listen 127.0.0.1:7101 --data .flower/cluster/node1
./target/release/flower --id 2 --listen 127.0.0.1:7102 --data .flower/cluster/node2
./target/release/flower --id 3 --listen 127.0.0.1:7103 --data .flower/cluster/node3

export FLOWER_ADMIN_TOKEN='replace-with-a-shared-operator-secret'
node sdk/cli.ts init \
  --members 1=127.0.0.1:7101,2=127.0.0.1:7102,3=127.0.0.1:7103

curl -fsS -H "Authorization: Bearer $FLOWER_ADMIN_TOKEN" \
  http://127.0.0.1:7101/raft/metrics

Use any reachable member as the entry address. Servers forward mutations and deployments to the current leader over pooled HTTP/2 and preserve request identities across elections. Queries and watches execute on the addressed replica; fresh reads wait for a quorum fence and local application. Metrics expose leadership for operators, not SDK routing:

export FLOWER_URL='http://127.0.0.1:7102'
node sdk/cli.ts deploy examples/orders.ts
# Or pass --url http://127.0.0.1:7102 on an individual command.

For separate hosts, use addresses peers can reach. --listen binds the HTTP socket; --advertise HOST:PORT sets the peer address when it differs. Advertised/member addresses omit http://. The SDK needs no leader discovery. If the entry node itself stops, select another reachable member; entry-node retries and watch reconnection remain application responsibilities.

Raft uses 50 ms heartbeats and a randomized 150–300 ms election delay. Its additional 300 ms leader lease means failure detection normally takes roughly 450–600 ms before election, storage, and scheduling overhead. For slower networks or disks, set FLOWER_RAFT_HEARTBEAT_MS, FLOWER_RAFT_ELECTION_MIN_MS, and FLOWER_RAFT_ELECTION_MAX_MS consistently on all members. Values are integer milliseconds with heartbeat < minimum < maximum. Shorter timeouts can cause unnecessary elections under load; acknowledgements and strong reads still require a quorum.

Internal Raft RPCs use pooled HTTP/2 prior-knowledge connections on these same ports. Both HTTP versions enforce the same operator token and peer identity checks. HTTP/2 does not encrypt traffic. Mixed builds must satisfy the same checked compatibility contract.

Know what an acknowledgement means

Votes, logs, application state, receipts, and snapshots persist in redb. Mutation replies wait for quorum-durable Raft commitment and local atomic application. Application records, receipts, membership, and applied-index metadata update together in a deferred redb checkpoint transaction; each apply no longer performs an additional fsync. The next immediate log, vote, snapshot, or purge transaction also makes earlier checkpoints durable. Concurrent methods can share a durable group commit while keeping separate results, revisions, and rollback boundaries. Under sustained load, the writer prepares the next group while the current group commits; reads and watches see only committed values. An uncertain commit discards its prepared successor and requires retries with the same request IDs. Deployment cutovers and scheduled maintenance run between drained pipeline windows. Default online deployment preparation runs outside the writer lane; explicit blocking preparation holds that lane.

With three members, one failed node leaves a quorum. A minority cannot acknowledge writes or serve strong reads. After startup recovery, explicit replica-local methods can still read their local committed snapshot, including older application code and aliases. Restart a stopped node with the same ID and data directory; it recovers and catches up through logs or a snapshot. If durable logs extend beyond its recovered applied position, one serving barrier covers that node and all its logical partitions until a quorum-confirmed fence is applied locally. Even replica-local application reads can be unavailable during this recovery; peer RPCs and diagnostics remain available. Flower never treats an uncommitted durable tail as committed work. Do not initialize it again or reuse its data directory under another ID.

Application records and receipts use separate redb tables, with applied-index and revision metadata in the same atomic checkpoint transaction. A crash can roll back a deferred checkpoint; the quorum-durable Raft log reconstructs committed changes and receipts without rerunning TypeScript. This uses the existing redb storage engine, not a separate WAL or general background checkpoint scheduler. Snapshots contain the complete application state. Raft builds them from immutable roots outside the apply lock, uses temporary files and durable checksummed chunks, and fences publication against a newer installation. Live application state still resides in memory; snapshot streaming does not make Flower a disk-backed query engine. Snapshots trigger on log count plus configurable applied bytes and nonidle age. Byte/age triggers observe a cooldown measured from build cost; count triggers can bypass it, and age restarts with the process. Inspect the snapshots object in GET /admin/resources and the snapshot scheduling settings. Flower is unreleased: APIs and stored/wire formats may change without backward compatibility or a migration path. The current server uses redb 4.3; unsupported older redb formats are rejected. Supported checkpoint metadata migrates forward at open and marks the old layout unreadable by older binaries. Downgrading that directory is unsupported. The checkpoint recovery change also updates the peer compatibility contract, so pre-change and current builds cannot mix; use a coordinated upgrade for this change.

Set capacity for your machine

Query work runs independently of the ordered writer lane. These settings are available in ordinary builds; they do not need an experimental feature flag. Worker and queue counts must be positive and representable by the runtime. Bigger queues retain more waiting work; they do not create CPU capacity.

Speculative writer preparation starts with pairs, expands after a fully reusable wave, and exponentially reduces probe frequency under repeated conflicts. While a staged target graph is maintained, speculative probes are bypassed; serial batching and grouped commits remain available. See deployment tuning. A successful probe restores growth; the configured queue capacity bounds the cooldown. Set FLOWER_WRITER_PREPARATION_WORKERS=1 for serial writer preparation while preserving query parallelism. Consecutive queued public calls in a serial stretch share one blocking dispatch, including during adaptive conflict cooldowns. Each call still gets its own isolated callback heap, guards, rollback boundary and receipt; authenticated applications and deployments keep their ordinary paths. Idle mutations start immediately. Adaptive batching balances queued work against measured preparation and commit costs; later arrivals can join a successor while its predecessor commits. Admission, serialized-byte, and preparation-time budgets still bound each group. When the local unapplied log or a voting quorum’s replication backlog exceeds the normal one-group overlap, the writer finishes its current commit before preparing a successor. A lone slow replica does not impose this gate; every majority in joint membership does. Remote matched positions are replication measurements, not follower application measurements.

Fresh reads can share quorum work: each replica batches fence requests, and the leader batches quorum proofs. A request joins only a proof that starts after it arrived, never a cached or already-running proof. One deadline covers queueing, the proof, and local application: FLOWER_READ_TIMEOUT_MS, ten seconds by default. Replica-local methods bypass these proof queues after the shared startup recovery fence has completed; writer barriers are unchanged.

Watches have no fixed total-count limit. Per-stream buffering and evaluator admission remain bounded; connections, memory, CPU, and the operating system still determine practical capacity. Measure with your own query cost and update rate.

Choose execution and transport budgets

Defaults are starting points. Set these environment variables before starting a node; settings are validated and cached at startup. Use matching policy on cluster members. Increasing an allowance spends memory or waiting time; it does not weaken transaction isolation, quorum requirements, or durable acknowledgement.

Serial mutation batches reuse resident instances after an exact reset. Linux/macOS copy all pages ever written; untouched pages stay read-only and unchanged. Other platforms use full copies. FLOWER_WASM_DIRTY_PAGES=0 selects full copies; FLOWER_WASM_RECYCLE=0 disables resident reuse. Idle instances are confined to one batch/thread and share the configured byte allowance and half the process’s instance slots. This trades memory and available slots for less setup work; host rights and transaction memory charges are renewed for every callback. See the complete limits and platform tradeoffs.

Allowances are positive decimal integers, except FLOWER_WASM_RECYCLE_BYTES, which also accepts zero. Source and result buffers must fit the guest’s signed 32-bit length with a trailing NUL. Guest memory must fit Wasm32’s 4 GiB address space; bundle size cannot exceed guest memory, and results cannot exceed either guest or Rust memory. Host allocations, pool counts, and durations must be representable by their runtime. On 64-bit hosts, each pooled Wasm memory reserves a 4 GiB virtual address range for hardware bounds checks. Reserved address space is not physical RAM; the configured live-memory allowance and pool growth maximum remain in force. These are not process RSS caps: stored state, compiled code, caches, and native thread stacks have separate costs. JSON and graph depth checks and native-stack guards remain structural safeguards.

HTTP headers have separate configurable budgets: FLOWER_HTTP1_MAX_HEADERS, FLOWER_HTTP1_MAX_BUFFER_BYTES, and FLOWER_HTTP2_MAX_HEADER_LIST_BYTES. They do not change body or result allowances; see the complete settings reference below.

Consensus separately bounds commands (FLOWER_TRANSACTION_MAX_BYTES, 32 MiB by default) and complete RPC envelopes (FLOWER_RPC_MAX_BYTES, 64 MiB). Batches account for their actual wrapper and separators. The consensus settings reference lists read/commit/peer deadlines, snapshot chunks and retention, Raft timing, and cross-budget validation. Raise the application request, evaluator, consensus, and client transport allowances together when increasing payload sizes.

A practical operator loop

1. Check the process. GET /health is public liveness only. It does not prove quorum or read database state.
2. Check consensus. Authenticated GET /raft/metrics shows leader, membership, and replication progress. Look at server logs during elections or storage errors.
3. Use a reachable entry node. Servers route writes to the current leader. Preserve request IDs on uncertain retries; if the entry process itself stops, choose another reachable member. Distribute queries and watches through read endpoints.
4. Deploy deliberately. Code, public aliases, maintenance registration, and recomputed values activate atomically. Use the staged lifecycle for durable graph preparation and independent page tuning to control the rebuild/write tradeoff. A rejected activation leaves the previous application active; source transformations remain explicit.
5. Preserve node data. Use persistent, separate directories. Use the membership API and rolling-upgrade procedure: learners catch up before promotion, and every peer checks the wire/state/guest compatibility contract. Roll compatible builds one member at a time. Incompatible contracts cannot communicate; a stopped cluster alone does not make its data format compatible.

Put the goblin kitchens under load.

Oven timers bake pizzas. Drones claim leased deliveries. A dragon can take out the leader. The audit counts every pizza and coin.

examples/goblin-pizza.ts exercises methods, derived statistics, timers, worker leases, receipt replay, and fencing. Its benchmark starts fresh local processes and never touches an existing cluster.

npm ci
npm run build
cargo build --release --bin flower
npm run bench
cargo build --release --bin flower --bin flower-bench-driver
npm run bench:stress

# Customize the base command, with one copy of each flag:
npm run bench -- --duration 30 --concurrency 16 --workers 4 \
  --max-orders 64 --drain 180 --json bench/results/custom.json
npm run bench -- --help

Watch the kitchens come alive

The live dashboard gets the selected tenant’s stores, orders, oven timers, drone leases, and rankings from one watched pizza.dashboard({tenant}) value. Switching tenants closes the old stream; new streams rotate across replicas. This replica-local view can lag without a bound, and reconnecting may show an older revision. The browser reconstructs deltas; it does not poll separate panels.

cargo build --release --bin flower
npm run demo:pizza
# Optional: fixed port, paused arrivals, stop automatically after two minutes
npm run demo:pizza -- --port 3030 --paused --duration 120

Open the printed local URL. The launcher creates a fresh three-node Rust/QuickJS cluster with three tenants, two stores per tenant, and tenant-scoped delivery workers. Place orders, tip goblins, pause arrivals, or test a leader crash. Workers continue when arrivals pause. During a disconnect, the board marks its last received replica state as stale and reconnects with a new snapshot. Countdown labels use approximate local time; database methods decide actual expiry.

Store references are [tenant, store]; canonical composite keys let tenants and stores reuse local order IDs. Each tenant has its own queue and derived leaderboard. Store summaries stay materialized; rankings compute when read, keeping whole-ranking work off the mutation path. pizza.shop.local([tenant, store]) permits lag and is the benchmark’s default preview; use --read-consistency fresh for a fresh-read run. pizza.shop([tenant, store]) and the group-wide pizza.world(null) audit remain fresh. Mutations check current stock and leases on the leader. This demo shows logical tenant separation, not caller authentication or tenant authorization.

Order creation is capped at 160 attempts, including button clicks; workers continue polling until shutdown. The board shows the latest 120 orders, with totals for the whole shift. Ctrl+C stops owned processes and removes their temporary data; the demo never attaches to an existing database.

The base command starts one three-replica group. The stress preset measures several independent groups, each with its own tenants, stores, customer loops, workers, and leader failure. See the benchmark guide for the current preset and options. Reports are written as JSON and self-contained HTML under bench/results/.

LATEST MEASURED RUN · 2026-09-24

A busy day in the garden.

Global customer calls / s

89,127

Customer p99

247.2 ms

Independent Raft groups

8 × 3 replicas

Replica-local reads: lag is allowed. 70% reads / 30% mutations · HTTP/2 · 60.1 measured seconds.

Run passed. 8/8 group audits passed. 8 injected leader failures; quorum recovery 556–745 ms.

Apple M5 Pro; all replicas and load generators share one machine. Completed customer calls use the union measurement window; retries, worker traffic, and explicit replays do not inflate throughput. Reads may be stale; mutations and audits retain fresh checks.

Charts & every group →Raw measurements ↓Workload & reproduction →

See the current measurements and analysis for measured throughput, latency, recovery, profiling, and remaining bottlenecks. Reports retain workload settings, read consistency, binary identity, resource use, and audit results so comparisons have a concrete basis.

The separate CPU investigation records repeated before/after measurements, native CPU profiles, and a cold-start contention test. Its repeated-run averages are distinct from the single latest run above.

The workload is closed-loop: a customer waits for one operation before starting its next. Before the order cap, choices are 30% orders, 40% reads, and 30% tips; afterward, order choices become reads. All servers and the generator share one machine. Results describe this colocated workload; production capacity depends on the workload and deployment.

npm ci
cargo build --release --locked --bin flower --bin flower-bench-driver
npm run bench:stress
node scripts/publish-bench-results.mjs bench/results/latest.json
node scripts/publish-bench-results.mjs --check

Run these commands inside nix develop when using the pinned development toolchain. Finish builds and tests before measuring, and leave profiling disabled for a capacity run. Publication regenerates the aggregate report, each group’s HTML and raw JSON, and the summaries on this page and the homepage. The offline check verifies that those artifacts agree.

The current stress preset uses Rust customer drivers with Node controllers, eight independent groups, 512 customer loops per group, 16 query/cache-probe slots and 16 shared preparation slots per node, two Tokio async workers, a 1,024-request writer queue, and a 200 ms maximum adaptive preparation window. Writer preparation is serial for this conflicting hot-store workload; groups and reads remain parallel. Existing environment variables override preset values; inspect the report’s recorded runtime settings when comparing runs. These measured-run settings override the ordinary defaults above. Concurrency, workers, and order caps apply to each group. Global customer throughput divides completed work across all groups by their combined measurement window; each group also has its own result and audit. Global latency percentiles come from merged observations, not averages of group percentiles.

Compare HTTP transports

The base benchmark command uses HTTP/1.1 for method calls by default. Add --http2 for pooled h2c method traffic; setup and leader discovery keep their ordinary transport. Raft peer traffic uses h2c in either case.

npm run bench -- --http2 --duration 30 --concurrency 32 --workers 4 \
  --max-orders 96 --chaos --json bench/results/http2.json

The report records the selected method transport. Keep workload settings and the binary the same when comparing protocols. The benchmark preserves request IDs across its retries; the HTTP/2 adapter itself does not replay uncertain operations.

Profile before tuning

On macOS, sample Rust/Wasmtime stacks against the same mixed load. Run without a leader crash for an easier first comparison.

CARGO_PROFILE_RELEASE_DEBUG=1 cargo build --release --bin flower
npm run bench -- --duration 30 --concurrency 32 --workers 4 \
  --max-orders 96 --drain 180 \
  --cpu-profile bench/results/profile-mixed.sample.txt \
  --json bench/results/profile-mixed.json

# Batch timing for the same mixed workload (one group).
FLOWER_PROFILE_STORAGE=1 FLOWER_PROFILE_EVALUATOR=1 \
node bench/profile-mixed.mjs --http2 --duration 30 --concurrency 128 \
  --workers 4 --max-orders 96 --json /tmp/flower-mixed-profile.json

The native profiler uses /usr/bin/sample on the initial leader. These are thread-stack observations, not CPU percentages or TypeScript source attribution. Profiling adds overhead; repeat without it for throughput claims. Reports retain the raw sample, configuration, binary identity, phase timings, and correctness results.

The mixed-workload diagnostic writes an additional *-groups.json with preparation, batch-fill, and commit timings. Optional storage and evaluator instrumentation adds stage detail. These phases can overlap; do not sum group lifetimes as elapsed wall time. Timings describe successful groups, with failed and dropped records counted separately.

For detailed request, writer, evaluator, storage, and Raft timing, node bench/profile-otel.mjs accepts the same workload flags and writes a separate local HTML report using sampled traces and unsampled metrics. See the OpenTelemetry guide for configuration, propagation, timing semantics, and capture limits. Reporting is off by default; instrumented throughput is a diagnostic measurement. The profiling commands also cover Instruments CPU samples, fixed-offered-rate CPU comparisons, and simultaneous cold-bundle queries.

Rust owns coordination; every application callback starts from pristine QuickJS/Wasm state, using a new instance or exactly restored resident storage. See the current measurements for the workload, useful throughput, and remaining writer bottleneck.

Run the checks

npm run check
cargo test
cargo build
node tests/e2e.mjs
node tests/e2e-http2.mjs
node tests/e2e-watch.mjs

The benchmark fails on unexpected logical errors, unfinished drain, invalid fencing/replays, or failed business invariants. Requested profiling must also finish successfully. Throughput is reported as a measurement. Ctrl+C stops owned processes and removes temporary data; --keep-data retains it for inspection.

Know the current growing edges.

Flower’s guarantees and operational limits should both be visible.

Each logical database has an ordered writer; named partitions share their physical group’s Raft log while preparing independently. Bounded speculative preparation validates candidates before ordered commitment. Evaluations share immutable record trees and copy changed paths. Normal storage applies write changed records and receipts; snapshots still contain the complete state. Live application state remains in RAM. Retry retention is opt-in, and transaction history needs explicit safe closure and collection. Application jobs, timers and other completed records need their own retention policy. The hot small-dataset benchmark does not establish large-state capacity.

Start with one method.

Then let its dependencies grow.

Back to the quick start ↗

Let the garden grow.

Named partitions keep their data, code, timers, leases and retry history together while moving between Raft groups. Register already initialized groups, then call client.resize(groupIds); a durable plan moves one partition at a time. The source keeps serving during durable base pre-copy; only the moving database pauses for final differences and ownership cutover. Large differences can fall back to a full frozen image, and all partitions still share physical resources. Group provisioning is separate, balancing uses partition counts, and a failed transfer rolls forward when its required groups recover.

See the complete partition API and operating contract for creation, movement, shrink/expand, ownership epochs, catalog availability, watch reconnection, and current memory/transaction limitations. Existing composite tenant keys are not automatically movable partitions.