#220
feat: add an opt-in concurrency cap for evals
Summary
Closes #136.
The start rate limiter spaces out container creation, but long-running evals can still accumulate hundreds of active sandboxes. Add --concurrency <n> to cap active eval attempts across all experiments and models in one CLI invocation. The default remains uncapped to preserve existing behavior.
- Share the limit in the runner, so it applies to Docker, Vercel, and library callers. Attempts retain their slots through retries and sandbox cleanup; queue time does not count against the eval timeout.
- With a cap, cleanup failure stops admission and cancels active attempts. Each cancelled agent gets up to 60 seconds to settle before the command exits nonzero. This can leave interrupted experiments without saved result summaries. Uncapped cleanup errors retain their existing handling.
- Fix the related start-rate reset bug that can leave later callers waiting indefinitely after a window fills exactly.
README and minor changeset included. Custom-agent cleanup requirements and the exported limiter/error types are documented.
Tests
Validated on Node v24.22.0:
npm run build,npm run lint, andgit diff --check: passed.npm test: 615 passed, 20 skipped. Includes built CLI flag handling and process exit, retries, cancellation, uncapped behavior, and cleanup failures.INTEGRATION_TEST=1 npm test -w packages/agent-eval -- src/integration.test.ts -t 'Docker concurrency': 3 passed. Real containers peaked at 4 uncapped, 2 with cap 2, and 1 during timeout cleanup with cap 1. No test-owned containers remained.
Docker tests use a local runner without agent credentials. Live Vercel and paid agent integrations were not run. The missing AbortSignal.any fallback is tested, but a full Node 18 run was not verified.