test: add an opt-in evaluation for Memory rule suggestions

Summary

This adds an offline evaluation that measures how well a model decides when to suggest a Memory rule with propose_memory_rule. It never runs in CI. Running it needs an explicit --run flag and an endpoint, a model and an API key from the environment.

Stacked on #2056.

What's included

• Labelled data in backend/tests/evals/memory_conventions/data/: dev.jsonl (1,391 items) and holdout.jsonl (416 items), in English and Chinese. Categories:
  • rules stated with and without a scope phrase;
  • a request mixed with a rule;
  • temporary, this-task-only requests;
  • updates and revocations of earlier rules;
  • rule-like text that comes from tool output, web pages, files, pasted text or "ok" replies, rather than from the user;
  • messages containing fake secrets;
  • ordinary messages.
• Runner run_eval.py:
  • Real request. Each item goes to an OpenAI-compatible endpoint with the agent guidance and the tool, read from the app code (MEMORY_RULE_NOTICE and the schema MemoryToolkit.get_tools() returns). Changing the tool's wording changes what the eval sends.
  • Same checks as the app. Every proposal is checked the way the app checks it: kinds, scopes, citations, content length, secrets and home paths.
  • Scores. It reports recall, precision and false-suggestion rates per language and per category against release thresholds. It also reports hard failures: suggestions that cite a non-user message or contain a secret.
  • Scope. Scope accuracy is gated only on items where the user stated the scope. Items without a stated scope are reported separately.
  • Output. Responses and summaries go only to a gitignored output/ folder. The API key is never printed or written.
• Unit tests for the runner: 43 tests. They need no network or key.

How to run

cd backend
EIGENT_MEMORY_EVAL_BASE_URL=https://... EIGENT_MEMORY_EVAL_MODEL=... \
EIGENT_MEMORY_EVAL_API_KEY=... \
uv run python -m tests.evals.memory_conventions.run_eval --run

To score recorded responses again without network access, use --dry-run --responses <file>.

Testing

• pytest tests/app/lightweight_memory: 233 passed, including the runner's 43 tests.
• pytest tests/evals --collect-only: collects nothing, so CI never runs a model.
• ruff check and ruff format --check (0.14.3, as pinned by pre-commit) pass, and so does git diff --check.
• A dry run with answers built from the labels passes on the dev set. No live model run is included in this PR.