Engine: binary and multipart bodies, and large artifacts (blocks ~30 hosted endpoints)

Open

xinyunwopened 10d ago

Problem

The engine moves JSON only. The transport reads every response with response.text() (engine/transport.ts), and request bodies are JSON. Four things that agents need from media providers do not fit:

Binary responses. Audio, images and video come back as bytes. Read as text, they are corrupted.
File inputs. Some vendors accept only a multipart file upload, not a URL.
Multipart form encoding. Some vendors take only multipart/form-data even when every field is text.
Large artifacts. Even when a vendor returns base64 inside JSON, a multi-MB video does not belong in run output or fn state.

This is already blocking work in this repo:

contextdev#parse is not ported because its request body is raw file bytes (add-connector-contextdev D6, tasks 7.2).
On #68 (Bright Data), the review asked to drop data_format: "screenshot" because the PNG would come back corrupted.

Hosted endpoints that cannot port until the engine handles these

provider	endpoints	gap
elevenlabs	text-to-speech, text-to-dialogue, sound-generation, music	audio response
elevenlabs	speech-to-text	multipart form encoding only: the audio is a URL (`source_url`) and the response is JSON, but the upstream accepts only `multipart/form-data`
elevenlabs	forced-alignment, audio-isolation, speech-to-speech	audio file input (multipart), plus audio output for the last two
fishaudio	text-to-speech, text-to-speech/timestamps, speech-to-text, voices/clone	audio in or out (hosted only; shapes inferred from the catalog)
topaz	11 `/image/*` endpoints	multipart image upload
gemini	5 video-generation endpoints	URL inputs inlined as base64; the MP4 returns inline as base64
smolmachine	download-file and file upload	file bytes
contextdev	parse	raw file request body

That is about 30 endpoints across 5 providers, plus the two items above. (Topaz's 12 /video/* endpoints take a source URL and return a download link, so they port today.)

How monid-services handles it (for reference)

Inputs: the caller passes a public https URL. The platform fetches it server-side (SSRF-guarded, size-capped at 20 MB for relays) and sends the bytes upstream as the multipart file part (fetchBinaryInput).
Outputs: binary output is stored in-activity and offloaded to the workspace file system. The run returns a link, not bytes (offload-run-media-outputs).

Possible shapes (for discussion)

A binary-aware transport: utils.http returns bytes when the content type is not JSON, plus a multipart request-body form. The bytes then go either to the host via an artifact port, or are base64-encoded with a size cap.
An artifact port on the host ABI (utils.artifacts.put(bytes, contentType) → url), so a fn hands off large outputs instead of inlining them. The hosted catalog already does this, so the port maps onto existing infrastructure.
A URL-input relay in the engine (utils.fetchInput(url, {maxBytes})), so connectors keep the "public URL in" contract the hosted catalog uses.

Any of these is a transport and hook-ABI change: an ENGINE_VERSION minor bump with fn_abi_since, plus a design.md decision.

Activity

No activity yet.