Run iterator builtins and their callbacks as bytecode so they can suspend

Problem

Several builtins drive an iterator, or call a user callback, from inside a single Rust call. A Rust builtin cannot yield to the host part-way through, so two things fail inside them that work in a plain for loop:

1. Loading an unread file. Content from open() lives on the host until a ReadText/ReadBytes call. Only the ForIter opcode (via the BufferLoad effect) and next() (via FileNext) can issue that load, because the VM knows how to finish those instructions once the reply arrives. Any Rust consumer of the iterator protocol raises NotImplementedError: iterating an unread _io.TextIOWrapper is only supported by 'for' loops and next(); call seek(0) first to load it (limitations/open.md).
2. Suspending in a callback. Callbacks these builtins invoke run through the synchronous evaluate_function path, which runs a frame to completion. A callback that reaches an external function, an os function, a host method call, or an await raises NotImplementedError: <context>: ... is not yet supported in this context.

The TODO on unloaded_iteration_error in crates/monty/src/types/file.rs records the plan for the first group: once enumerate, zip, map, filter and sorted are bytecode, they iterate through ForIter and the problem goes away for them. This issue writes that plan down with every case it covers.

The bytecode approach

Move the loop out of Rust and into the VM, so that each iteration step is a ForIter and each callback invocation is an ordinary call opcode. Both of those already suspend and resume correctly: ForIter rewinds and re-dispatches after a BufferLoad, and a call frame can yield an OsCall or an external-function request and be resumed with the result. Nothing new is needed in the host protocol.

Two shapes are possible, and they are not exclusive:

• Compile the for-header forms specially. for i, x in enumerate(xs), for a, b in zip(xs, ys), for y in map(f, xs) and for x in filter(p, xs) can be compiled into the loop itself when the name resolves to the builtin, the way comprehensions already compile to for loops (limitations/open.md notes that list(x for x in f) works for exactly this reason). This makes those positions lazy as well, fixing the eager-vs-lazy divergence in limitations/builtins.md for the loop case: mutation of the source is observed, an infinite source is fine, and the callback runs as items are consumed.
• Define the builtin in Python. sorted(key=...), min/max with key, functools.reduce, iter(callable, sentinel) and the itertools adaptors that take a callable can be written as Python source compiled with the program, so their loop and their calls run in the VM. Monty has no generators, so these stay eager where they are eager today, but the callback becomes suspendable. The cost is bytecode-speed iteration where there is Rust today, so the hot paths (sorted without key, map/filter over a loaded list) should keep a Rust fast path and only fall back to the bytecode form when the source is an unread file or the callback is not a plain sandbox function.

Known cases this solves

Unread-file iteration (limitations/open.md):

• enumerate(f)
• zip(f, ...) and zip(..., f)
• map(g, f)
• filter(p, f)
• sorted(f)

Callbacks that cannot suspend today, each raising NotImplementedError on an external function, os call, host method or await:

• sorted(key=...) and list.sort(key=...) (limitations/open.md, limitations/copy.md)
• min(key=...) / max(key=...) (limitations/open.md)
• the function passed to map() / filter() (limitations/open.md, limitations/functools.md)
• iter(callable, sentinel) (limitations/iter.md)
• functools.reduce(), including through a partial (limitations/functools.md)
• itertools.takewhile, dropwhile, filterfalse, starmap, accumulate and groupby (limitations/itertools.md)

Eager-vs-lazy divergences in the loop position (limitations/builtins.md), fixed by the for-header form:

• type(enumerate(x)) aside, a side-effecting callable in map/filter runs per consumed item rather than all at the call
• map(f, itertools.count()) and other infinite sources terminate when the loop breaks
• dict, set and deque mutation during for ... in enumerate(d) raises as in CPython

Known cases this does not solve

Rust consumers with no callback and no loop of their own stay on the call-boundary replay that LoadThenCall provides (introduced for csv.reader(f) / csv.DictReader(f) in #1018, from the same TODO): yield the load with the call stashed, replay the call on resume. That covers list(f), tuple(f), set(f), ''.join(f), sum(f), any(f), all(f), min(f) and max(f). These could also be given the bytecode treatment, but they are hot and have no callback, so the replay is the cheaper fix.

Implicitly dispatched dunders and factory hooks are a separate synchronous context and are out of scope here: __repr__/__str__, __eq__/__hash__/__index__, __iter__/__next__/__contains__, an external-function __init__ (limitations/classes.md), defaultdict's default_factory (limitations/collections.md), dataclass default_factory and __post_init__ (limitations/dataclasses.md), __copy__/__deepcopy__ (limitations/copy.md), str.format attribute access (limitations/format.md), and the write() target of csv.writer.writerows() (limitations/csv.md).

Pointers

• crates/monty/src/types/file.rs: unloaded_iteration_error and its TODO; needs_buffer_load / buffer_load_call
• crates/monty/src/os_dispatch.rs: PostConversionEffect::{BufferLoad, FileNext, LoadThenCall}
• crates/monty/src/bytecode/vm/mod.rs: the ForIter arm that issues BufferLoad, resume_load_then_call
• crates/monty/src/builtins/{enumerate,zip,map,filter,sorted,min_max}.rs: the eager Rust implementations
• crates/monty/src/bytecode/compiler.rs: comprehension compilation, the model for the for-header form

🤖 Generated with Claude Code