#20

RAI-303 (draft, do not merge): voice rules in the prompts and the style eval

Drafttobi-techywants to merge intofrom

Stacked on #19. Linear: RAI-303. DRAFT: it does not meet its own bar, so it is not merged.

What is here

  • A "How you write" rule 9 in CORE_RULES, plus voice lines in the Collector, Mimi chat (now named for the distributor), Order and Credit skills.
  • Delivery thanks template: "Thanks for confirming. Delivery X is now marked as received." flag_text was factored out.
  • mimi/api/style_eval.py and eval/style/: 20 dev cases, 70 held-out cases (46, then 12 fresh before round 2 and 12 more fresh before round 3) and 8 hand-labelled calibration replies. Each reply is checked in code (dashes, banned phrases, figures unchanged, figures grounded, length, exclamation marks, no Pidgin) and by a Muse Spark judge (natural, professional, clear, plain English, on point). A case passes only if all of these pass.

Results (held-out; tuned on dev only)

  • Round 1: dev 75%, held-out 74% (34/46). One invented figure, an eval fixture bug, now fixed.
  • Round 2: dev 90%, held-out 72% (42/58); the 12 fresh cases alone 67%. One banned phrase ("Glad to help"); the guard was fixed for it.
  • Round 3: dev 85%, held-out 74% (52/70); the 12 newest cases alone 67%.
  • Dashes, figure changes and invented figures were zero in rounds 2 and 3. The judge agreed with all 8 calibration labels every round.
  • On 7 held-out cases run on the old prompts with no guard: 3 to 4 of 7 passed, and 2 to 3 had em dashes.

Why it is below 90% Most misses are "on point" (an answer that leaves out a fact, such as the invoice number or the total next to the overdue figure) and "natural" (stiff openings, a bot-like note to the owner). A round-3 rule ("a polite ask") made things worse in places: it asked for payment on an account-details request. The judge is also strict on some code templates ("Delivery X is confirmed" counts as "system-written").

Not done The harness, context, credit and e2e caches were NOT re-recorded for these prompts (about N8,700), so CI evals on this branch miss their caches. Re-record only after the prompts pass. One held-out case (h3-col-pending-ref) was edited after recording: a real bank short name in its reference was replaced to satisfy test_no_real_banks, so it now misses its cache.

Activity

3
coderabbitai[bot]3d ago

IMPORTANT

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This pull request is a draft

Merging is unavailable until it is marked ready for review.