local(): idle session.stream() readers and each prompt cost time in proportion to session length

Summary

On the default Node.js store, local(), a long session gets slower and costlier as it grows:

1. An open session.stream() reader polls the session's whole chunk folder every 100 ms. It costs CPU in proportion to the session's length, while nothing happens: 14.7% of a core for one idle reader on a 25,100-chunk session, and 33% for two.
2. Every prompt reads the first byte of every chunk file to find the stream's length. A prompt was accepted after about 1.2 s on that session, against 50–90 ms on a 500-chunk one.
3. Each stream line is its own file. A real chat of 24,539 chunks held 2.1 MB of data in 98 MB of disk, at a 4 KB block per file.

The poll lives in @workflow/world-local, which local() bundles, and is already reported there as vercel/workflow#4830. We're filing here because the way libfx uses that store, one ever-growing stream per session, is what makes it costly for a chat app. We'd like to hear which path you'd prefer before we change anything on our side.

We hit this in fxbot, a self-hosted chat app on libfx. Each chat is one libfx session on local({ dir }), and the chat page holds session.stream(cursor) open while the chat is on screen. An installed app left open in the background on a long chat kept the idle server at about 16% of a core.

Environment

• libfx 0.0.16, which bundles @workflow/world-local 5.1.1. The installed durable.js is byte-identical to the v0.0.16 tag.
• Node.js 22.23.4, Debian 13, x64, ext4.

What we measured

With libfx alone, using the scripts below:

Every number is the process's own CPU over 20 s after the readers caught up, with nothing happening. The two-reader case splits as 14.8% user and 18.3% system. In fxbot, closing the reader brings the server back to 0% at once.

Reproduction

A mock model that answers each request with 500 small deltas, so each turn adds about 500 chunks:

// mock.mjs: node mock.mjs
import { createServer } from "node:http";
createServer(async (req, res) => {
  let body = ""; for await (const c of req) body += c;
  const { model } = JSON.parse(body || "{}");
  const chunk = (delta, finish = null) => `data: ${JSON.stringify({ id: "c", object: "chat.completion.chunk", model, choices: [{ index: 0, delta, finish_reason: finish }] })}\n\n`;
  res.writeHead(200, { "content-type": "text/event-stream" });
  res.write(chunk({ role: "assistant", content: "" }));
  for (let i = 0; i < 500; i++) res.write(chunk({ content: `w${i} ` }));
  res.write(chunk({}, "stop")); res.end("data: [DONE]\n\n");
}).listen(11630, "127.0.0.1");

The script grows a session to a number of chunks, opens idle readers, and measures its own CPU:

// repro.mjs: node repro.mjs <dir> <chunks> <readers>
import { createFxAgent } from "libfx";
import { local } from "libfx/durable-local";
import { readdirSync, existsSync, readFileSync, writeFileSync } from "node:fs";

const [dir, target, readers = "1"] = process.argv.slice(2);
const agent = createFxAgent({
  name: "repro", model: "mock-a", durability: local({ dir }),
  provider: { name: "mock", baseUrl: "http://127.0.0.1:11630/v1", models: {} },
  credentials: async () => ({ apiKey: "x" }),
});
const idFile = `${dir}/session-id`;
const id = existsSync(idFile) ? readFileSync(idFile, "utf8") : await agent.newSessionId();
writeFileSync(idFile, id);
const count = () => {
  const base = `${dir}/streams/chunks`;
  const name = existsSync(base) && readdirSync(base).find((n) => n.includes(id.replace("wrun_", "")));
  return name ? readdirSync(`${base}/${name}`).length : 0;
};
const session = agent.session(id, { model: "mock-a" });
while (count() < Number(target)) await session.prompt("go").result;
for (let i = 0; i < Number(readers); i++) {
  const reader = session.stream(0).getReader();
  (async () => { while (!(await reader.read()).done); })();
}
await new Promise((r) => setTimeout(r, 5000));
const before = process.cpuUsage(), start = Date.now();
await new Promise((r) => setTimeout(r, 20000));
const used = process.cpuUsage(before);
console.log(`${readers} idle reader(s) on ${count()} chunks: ${((used.user + used.system) / ((Date.now() - start) * 1000) * 100).toFixed(2)}% of a core`);
process.exit(0);

To time a prompt, call session.prompt("go") on an existing session and time turn.accepted.

Where it comes from

Permalinks are at fx v0.0.16 and at @workflow/world-local@5.1.1, the version libfx pins.

• One chunk per line. ui.write turns each UI line into its own chunk (durable.js#L1337-L1342). Each text_delta and reasoning_delta is a line, and world-local stores each chunk as its own file.
• The idle poll. session.stream() (#L2978) reads through world.streams.get() (#L1361).
  • In world-local, get() subscribes to an in-process EventEmitter, and also starts a setInterval(…, 100) (streamer.ts#L854-L906).
  • Each tick, that interval runs listChunkFilesForStream (#L138): a readdir of the stream's folder, a map over every name, and a sort of every name. Then it walks every name against the set of chunks it has delivered.
  • The comment calls it the cross-process fallback. It runs for every reader, whether or not the writer is in-process, and can't be configured.
• The per-prompt read. ui.length() calls world.streams.getInfo() (durable.js#L1344-L1350). world-local's getInfo reads the first byte of every chunk file (streamer.ts#L624-L648). A prompt calls it to find where its viewer starts (durable.js#L2827, #L2858).
• pollMs isn't involved. local()'s pollMs: 250 (local.mjs#L26) drives the worker's re-read of the session log for steers and cancels during a turn (durable.js#L2267), not stream readers.

Related

• vercel/workflow#4830 (open): "Local World idle tail readers repeatedly scan all historical chunks in long streams". It has the same measurement at the Workflow level: about 13% of a core for one idle reader at 23,486 chunks.
• vercel/workflow#2797 and #2807 (closed): the earlier fix, which moved chunks into one folder per stream. That made each scan O(chunks in the stream) instead of O(chunks in the world).
• vercel/workflow#1739 (closed): where the poll came from, as cross-process streaming for local development.
• vercel/workflow#5061 (open): world-postgres sends a NOTIFY and runs a reader SELECT per chunk.
• vercel-labs/fx#1583 (open): deltas out of order with the bundled file World. It's fixed in world-local 5.2.0, and libfx still pins 5.1.1.
• @workflow/world-local's README describes it as a "Filesystem-based workflow backend for local development and testing". libfx's README lists local() as the Node.js default store.

Who it affects

Any app on local() that keeps a reader open on a long session: a chat page, a dashboard, a CLI following a session. The idle cost grows with the session's length times its open readers, and every prompt on a long session pays the getInfo read. Short sessions barely notice it. A chat app, where a conversation is one session that grows for weeks, feels it.

Options we're weighing on our side

We lean toward A and B now, and D later. A and B work around the store rather than fix it, though, and every other local() user would need the same.

Questions

1. Would you rather fix this in libfx, so every app gets it? Some possible ways:
   • share one stream reader per session
   • skip world-local's poll when the writer is in the same process: libfx knows when its own worker writes
   • track the stream's length without getInfo
   • write fewer chunks per turn, by coalescing deltas
2. Is local() meant for long-running production servers? Workflow describes world-local as being for local development and testing. If local() is meant for development too, a note in the README would help, along with what you'd recommend for production on Node.js: world() with world-postgres?
3. Do you plan to move to world-local 5.2.0? It fixes #1583, though not the poll.
4. Would a fix in world-local, for vercel/workflow#4830, be your preferred route, with libfx then bumping its pin?

We're happy to send a PR for whichever direction you prefer.