Batch embeddings by token budget, not just input count

Problem

Indexing a large corpus dies with:

Embedding API error (400): Requested 375703 tokens, max 300000 tokens per request (max_tokens_per_request)

embed() batched inputs only by count (MAX_BATCH = 2048), but the OpenAI-compatible /v1/embeddings endpoint caps each request by tokens (300k). Once the changed-section set embeds to more than ~300k tokens in a single array, every request 400s and the index never builds — there's no partial write, so lat search stays broken.

Fix

Add planBatches(), which splits inputs into requests bounded by both input count (2048) and estimated tokens (~250k, comfortably under the 300k cap; a ~4-chars/token estimate, no tokenizer dependency added). Order is preserved, so embed() still maps returned vectors back to inputs positionally. embed() now iterates planBatches(texts) instead of a fixed-stride slice.

Tests

New always-run planBatches unit tests (no API/replay needed): single small batch, token-budget split, no empty batches, and order/no-loss across the 2048 count cap. typecheck clean; tests/search.test.ts 13/13 green.

🤖 Generated with Claude Code