#137

[Agents] Add Mistral Vibe CLI agent with direct API and streaming-json transcript support

Openjoeyspagnoliwants to merge intofrom

Closes #90.

Adds Mistral Vibe as a new eval agent using direct API access via MISTRAL_API_KEY. The implementation follows the same pattern as the Cursor and Gemini agents, with a dedicated transcript parser that normalizes Vibe's --output streaming JSONL (OpenAI-compatible LLMMessage rows) into the canonical o11y schema.

Vibe CLI requires --prompt to enter programmatic mode (which auto-selects the auto-approve profile and bypasses tool-permission prompts that would otherwise hang in sandboxed environments), --max-turns to bound runaway agents (defaulted to 50, overridable per-experiment via agentOptions.maxTurns), and --output streaming to emit newline-delimited JSON per message for transcript parsing. We also pass --trust to silence Vibe's untrusted-workdir stderr warning and honor any project-level .vibe config — the interactive trust dialog is already skipped when --prompt is set (vibe/cli/entrypoint.py:204-206), so this is hygiene rather than a hang fix. The model is configured via the VIBE_ACTIVE_MODEL env var rather than a CLI flag — Vibe overrides any config field through VIBE_* env vars — and telemetry, OTEL exporters, and auto-update are all disabled the same way so the sandbox stays quiet.

Installation downloads the uv binary directly from GitHub releases (pinned to 0.11.15) via Node's built-in fetch and extracts with tar, rather than using upstream's curl | sh installer. This lets the same install chain work on both the Vercel sandbox (which has curl) and node:*-slim (which doesn't, and which the local Docker sandbox uses); both backends are exercised in local integration testing. uv tool install mistral-vibe then handles the Python 3.12 + Vibe install in one step. The install chain runs under set -ex so the truncated error tail identifies which stage failed if anything breaks. The default model is mistral-medium-3.5, which became Vibe's default in v2.9.1 (replacing the previous devstral-2 alias).

The parser maps Vibe's native tool names (read_file, write_file, search_replace, bash, grep, web_fetch, web_search, todo, task) to the canonical o11y tool union, extracts file paths, shell commands, and URLs into _extracted* keys, emits reasoning_content as a separate thinking event, skips harness-injected (injected: true) rows, and pairs role: 'tool' results with their originating tool calls via tool_call_id. The tool map has been verified against a captured streaming transcript from a real run; Vibe ships no glob or list_dir tool (models use bash for directory ops) and skill falls through to 'unknown' as a meta-tool with variable shape. The role: 'tool' success heuristic uses a prefix match against common error indicators (error:, failed:, traceback, etc.) so legitimate output mentioning those words mid-stream is not misclassified. Integration tests run end-to-end against the real Mistral API on the Docker sandbox.

Activity

9
benjamincanac5mo ago

@joeyspagnoli I made a similar PR https://github.com/vercel-labs/agent-eval/pull/101 a while back, let me know what improvements you made to see if I should close mine!

joeyspagnoli5mo ago

hi @benjamincanac ! a few differences between #101 and this PR:

  • Default model: mistral-medium-3.5 (Vibe v2.9.1 renamed it from devstral-2; the old alias still resolves via a _migrate shim).
  • Model selection: no --model flag exists; the model is set through VIBE_ACTIVE_MODEL (the VIBE_* env override pattern documented in entrypoint.py's argparse epilog). This PR also passes --trust (added v2.9.0, silences the workdir-trust stderr warning in programmatic mode), --max-turns to bound runaway loops, and VIBE_ENABLE_{TELEMETRY,OTEL,AUTO_UPDATE}=false to quiet the sandbox.
  • Install: curl | bash doesn't work on node:24-slim (no curl, no apt root), and Mistral's own install.sh shells out to curl too. This fetches the uv tarball directly via Node's built-in fetch (pinned to 0.11.15), then uv tool install mistral-vibe. Verified end-to-end on Docker for both arm64 and amd64 against the live API.
  • Parser: Vibe's --output streaming emits OpenAI-compat LLMMessage rows (role/content/tool_calls/reasoning_content/tool_call_id), not type-discriminated events. This PR's parser handles that schema, pairs role:"tool" results with their originating call via a per-tool_call_id pending Map (parallel tool calls work), skips injected:true harness rows, and emits reasoning_content as a separate thinking event. Tool map covers 9 of the 10 active builtins; skill maps to unknown because no canonical name fits (every parser in the repo defaults unmapped tools to unknown this way). ask_user_question/exit_plan_mode are auto-disabled in programmatic mode.