Durable UI stream: text deltas can arrive out of order with the bundled file World

Summary

With the leased file durability (world(local({ dir }).world, { leaseMs, ... })), a turn's text_delta events sometimes reach the turn view in the wrong order. One delta lands later than it should, so the text reads like "checmarks … together.k". We saw it in 1 of 130 turns that had a single-step answer. libfx 0.0.14 and 0.0.16 have the same code.

The root cause is in the file World that libfx bundles (@workflow/world-local, inside libfx/durable/local.mjs), so this may belong in vercel/workflow as well.

Cause

1. libfx's World backend batches the UI lines a worker pushes while a write is in flight. It then sends them with world.streams.writeMulti(runId, uiStream, chunks) when there is more than one (durable.js, ui.write: line 1054 in 0.0.14, line 1341 in 0.0.16).

2. The bundled file World's writeMulti names each chunk with a monotonic ULID, then writes all the chunk files at once (chunks.map(async ... await writeFile(...))). It emits each chunk's event in order, but only after that chunk's own file has landed.

3. The same World's reader, streams.get(runId, name, from), takes chunks two ways:

   • at open, it lists the stream's files and reads every one not yet seen, buffering events that arrive meanwhile;
   • live, it takes events, and every 100 ms it also polls the folder for files it hasn't seen.

   Both paths enqueue a chunk as soon as its file exists. With concurrent writes, chunk N+1's file can exist before chunk N's. The reader then enqueues N+1 from the listing or the poll, and N later from its event.

Repro

This uses the bundled World alone, with no agent. Run it from a package that depends on libfx 0.0.14: npx tsx world-order.mts <concurrent|sequential> <runs> <chunks>. It opens a reader on a fresh stream, writes numbered chunks, and counts reads that come back out of order.

import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { local } from "libfx/durable-local";

const mode = process.argv[2] ?? "concurrent";
const runs = Number(process.argv[3] ?? 200);
const count = Number(process.argv[4] ?? 64);
type Streams = {
  write(run: string, name: string, chunk: string): Promise<void>;
  writeMulti(run: string, name: string, chunks: string[]): Promise<void>;
  get(run: string, name: string, from?: number): Promise<ReadableStream<Uint8Array>>;
};
const files = local({ dir: mkdtempSync(join(tmpdir(), "world-order-")) }) as unknown as { world: () => Promise<{ streams: Streams }> | { streams: Streams } };
const world = await files.world();
let wrong = 0;
let firstWrong: Array<number> | null = null;
for (let r = 0; r < runs; r++) {
  const run = `wrun_01PROBE${String(r).padStart(18, "0")}`;
  const name = `ui-${r}`;
  const chunks = Array.from({ length: count }, (_, i) => `${i}\n`);
  await world.streams.write(run, name, "start\n");
  const reading = (async () => {
    const reader = (await world.streams.get(run, name, 0)).getReader();
    const seen: Array<number> = [];
    let text = "";
    while (seen.length < count) {
      const { value, done } = await reader.read();
      if (done) break;
      text += new TextDecoder().decode(value);
      const lines = text.split("\n");
      text = lines.pop() ?? "";
      for (const line of lines) if (line !== "start") seen.push(Number(line));
    }
    void reader.cancel().catch(() => {});
    return seen;
  })();
  if (mode === "concurrent") await world.streams.writeMulti(run, name, chunks);
  else for (const chunk of chunks) await world.streams.write(run, name, chunk);
  const seen = await reading;
  const ordered = seen.every((n, i) => n === i);
  if (!ordered) {
    wrong += 1;
    firstWrong ??= seen.slice(0, 16);
  }
}
console.log(JSON.stringify({ mode, runs, chunks: count, outOfOrder: wrong, firstWrong }));
process.exit(0);

A burst that spans the reader's 100 ms poll reorders most of the time, and only with concurrent writes. A turn's batches are small, which is why a real turn reorders rarely.

Possible fixes

Either one is enough:

• In the file World, write writeMulti's chunks one after another, or write them all and only then make them visible, with a rename or a manifest.
• In the file World's reader, enqueue chunks only in id order, so a chunk waits until every chunk before it has been seen.

libfx could also put a sequence number on each UI line, so a turn view can order lines itself on any World.

What we do meanwhile

We build a turn's reply from the provider's own response stream instead of the text_delta events.