Closes #90.
Adds Mistral Vibe as a new eval agent using direct API access via MISTRAL_API_KEY. The implementation follows the same pattern as the Cursor and Gemini agents, with a dedicated transcript parser that normalizes Vibe's --output streaming JSONL (OpenAI-compatible LLMMessage rows) into the canonical o11y schema.
Vibe CLI requires --prompt to enter programmatic mode (which auto-selects the auto-approve profile and bypasses tool-permission prompts that would otherwise hang in sandboxed environments), --max-turns to bound runaway agents (defaulted to 50, overridable per-experiment via agentOptions.maxTurns), and --output streaming to emit newline-delimited JSON per message for transcript parsing. We also pass --trust to silence Vibe's untrusted-workdir stderr warning and honor any project-level .vibe config — the interactive trust dialog is already skipped when --prompt is set (vibe/cli/entrypoint.py:204-206), so this is hygiene rather than a hang fix. The model is configured via the VIBE_ACTIVE_MODEL env var rather than a CLI flag — Vibe overrides any config field through VIBE_* env vars — and telemetry, OTEL exporters, and auto-update are all disabled the same way so the sandbox stays quiet.
Installation downloads the uv binary directly from GitHub releases (pinned to 0.11.15) via Node's built-in fetch and extracts with tar, rather than using upstream's curl | sh installer. This lets the same install chain work on both the Vercel sandbox (which has curl) and node:*-slim (which doesn't, and which the local Docker sandbox uses); both backends are exercised in local integration testing. uv tool install mistral-vibe then handles the Python 3.12 + Vibe install in one step. The install chain runs under set -ex so the truncated error tail identifies which stage failed if anything breaks. The default model is mistral-medium-3.5, which became Vibe's default in v2.9.1 (replacing the previous devstral-2 alias).
The parser maps Vibe's native tool names (read_file, write_file, search_replace, bash, grep, web_fetch, web_search, todo, task) to the canonical o11y tool union, extracts file paths, shell commands, and URLs into _extracted* keys, emits reasoning_content as a separate thinking event, skips harness-injected (injected: true) rows, and pairs role: 'tool' results with their originating tool calls via tool_call_id. The tool map has been verified against a captured streaming transcript from a real run; Vibe ships no glob or list_dir tool (models use bash for directory ops) and skill falls through to 'unknown' as a meta-tool with variable shape. The role: 'tool' success heuristic uses a prefix match against common error indicators (error:, failed:, traceback, etc.) so legitimate output mentioning those words mid-stream is not misclassified. Integration tests run end-to-end against the real Mistral API on the Docker sandbox.
Activity
9hi @benjamincanac ! a few differences between #101 and this PR:
- Default model:
mistral-medium-3.5(Vibe v2.9.1 renamed it fromdevstral-2; the old alias still resolves via a_migrateshim). - Model selection: no
--modelflag exists; the model is set throughVIBE_ACTIVE_MODEL(theVIBE_*env override pattern documented inentrypoint.py's argparse epilog). This PR also passes--trust(added v2.9.0, silences the workdir-trust stderr warning in programmatic mode),--max-turnsto bound runaway loops, andVIBE_ENABLE_{TELEMETRY,OTEL,AUTO_UPDATE}=falseto quiet the sandbox. - Install:
curl | bashdoesn't work onnode:24-slim(no curl, no apt root), and Mistral's owninstall.shshells out to curl too. This fetches theuvtarball directly via Node's built-infetch(pinned to0.11.15), thenuv tool install mistral-vibe. Verified end-to-end on Docker for both arm64 and amd64 against the live API. - Parser: Vibe's
--output streamingemits OpenAI-compatLLMMessagerows (role/content/tool_calls/reasoning_content/tool_call_id), not type-discriminated events. This PR's parser handles that schema, pairsrole:"tool"results with their originating call via a per-tool_call_idpending Map (parallel tool calls work), skipsinjected:trueharness rows, and emitsreasoning_contentas a separate thinking event. Tool map covers 9 of the 10 active builtins;skillmaps tounknownbecause no canonical name fits (every parser in the repo defaults unmapped tools tounknownthis way).ask_user_question/exit_plan_modeare auto-disabled in programmatic mode.
@joeyspagnoli I made a similar PR https://github.com/vercel-labs/agent-eval/pull/101 a while back, let me know what improvements you made to see if I should close mine!