Results under MAX_FRAME_LEN that exceed the host decode budget kill the session as a protocol violation

A sandbox can legitimately return a value whose frame is well under MAX_FRAME_LEN, and the host still rejects it as a protocol violation. When that happens the worker is discarded and the caller's session is killed. If the same value is larger, so that its frame exceeds MAX_FRAME_LEN, the child reports a clean RuntimeError and the session keeps working. So a result that is too big for the host is handled differently depending on which of the two size limits it crosses first.

Commit: 4246aa75724b133e55f8a82e2cf5dfc0120e62e8 (Linux x86_64, release build, make dev-py-release)

Reproduction

from pydantic_monty import Monty

with Monty() as pool:
    for n in [8_000_000, 10_000_000, 15_000_000, 30_000_000]:
        with pool.checkout() as session:
            try:
                v = session.feed_run(f'list(range({n}))')
                outcome = f'ok, len={len(v)}'
            except Exception as e:
                outcome = f'{type(e).__module__}.{type(e).__qualname__}: {e}'
            try:
                after = repr(session.feed_run('1 + 1'))
            except Exception as e:
                after = f'{type(e).__qualname__}: {e}'
            print(f'n={n:,}: {outcome}\n    next feed -> {after}')

Output (default ReplConfig, no resource limits):

n=  8,000,000  ok, len=8000000
               next feed on same session -> 2
n= 10,000,000  builtins.RuntimeError: monty worker protocol error: invalid payload from worker: failed to decode Protobuf message: MontyNode.kind: Complete.values: ChildEvent.kind: frame exceeds decode memory budget
               next feed on same session -> RuntimeError: this checkout has already been finished
n= 15,000,000  builtins.RuntimeError: monty worker protocol error: invalid payload from worker: failed to decode Protobuf message: Complete.values: ChildEvent.kind: frame exceeds decode memory budget
               next feed on same session -> RuntimeError: this checkout has already been finished
n= 30,000,000  pydantic_monty.MontyRuntimeError: RuntimeError: result frame of 326829555 bytes exceeds the maximum of 268435456 bytes
               next feed on same session -> 2

The same thing happens through monty-pool from Rust: Checkout::feed returns PoolError::Protocol("invalid payload from worker: … frame exceeds decode memory budget").

Frame sizes, from encoding the same Complete with monty_proto::encode_to_capped_vec and decoding it with decode_frame:

Expected vs. actual

• Expected: a value the host can't accept is reported the same way as an over-length frame, as a session-preserving runtime error. At minimum it should not be classified as a misbehaving worker.
• Actual: anything between roughly 32–36% and 100% of MAX_FRAME_LEN (for a list of small ints) becomes PoolError::Protocol. The worker is discarded and the session is lost, even though the worker did exactly what it was asked to do.

What the code does (verified)

• The child checks only the encoded length. In monty-proto/src/worker.rs#L434-L441, FrameTooLarge on a result becomes an ExcType::RuntimeError event, and that is what produces the clean n=30M row.
• The host decodes under a second, independent ceiling. frame.rs#L26-L33 defines DEFAULT_MAX_DECODE_BYTES = 4 * MAX_FRAME_LEN, and decode_frame applies it. A list of small ints charges roughly 11–13x its wire size against that budget (it fits at 84.8 MB and not at 95.8 MB), so it runs out at about 32–36% of MAX_FRAME_LEN. (PR #883 hit the same amplification for a dag-shaped bench frame.)
• The host treats every decode failure as worker misbehaviour. checkout.rs#L1689-L1696 ("a decode failure means the frame arrived intact but its payload was garbage … the worker misbehaved") routes FrameError::Decode to protocol_violation, which discards the worker.

Hypotheses (not verified)

• The same split probably applies to other child → host values decoded under the budget, such as large FunctionCall/OsCall arguments, but I only reproduced it with Complete.
• With limits={'max_memory': ...} set low enough, the sandbox would fail before reaching this size. That means the default (unlimited) configuration is the one exposed.

Related but separate: #672 asks for these ceilings to be configurable. This issue is about the mismatch between the child's and the host's limits at their current defaults, and about how the host classifies the failure.