#164
Issues/snarktank/ralph/#164

RFC: Generator-Evaluator Dual-Agent Architecture (Harness Mode)

Openm18897829375opened 4mo ago

Motivation

The current Ralph is a highly effective single-agent loop. After extensive real-world use and battle-testing (especially on Windows/MSYS2), I've developed a significant enhancement: a Generator-Evaluator dual-agent architecture that adds contract negotiation, phase discipline, and 4-dimension quality scoring.

This RFC proposes adding "Harness Mode" as an optional architecture alongside the existing simple mode.

Key Concept

In Harness Mode, two specialized AI agents collaborate per story — Generator (builds, never judges) and Evaluator (judges, never builds):

  1. Contract Negotiation — Generator proposes scope → Evaluator reviews → back-and-forth until locked
  2. Build — Generator reads locked contract → implements → typecheck/lint/test → commits
  3. Evaluate — Evaluator tests in browser (Playwright MCP) → scores on 4 dimensions → writes evaluation.json
  4. Retry or Advance — Pass all thresholds → next story. Fail → feedback to Generator → retry

Proposed Changes

New (4 files): generator-prompt.md, evaluator-prompt.md, contract.json.example, AGENTS.md Modified (1 file): ralph.sh — purely additive, run_simple_mode() untouched

Default mode remains simple — fully backward compatible:

./ralph.sh                           # simple mode (unchanged)
./ralph.sh --mode harness --tool claude  # new harness mode

Questions

  1. Is the dual-agent architecture direction aligned with your vision?
  2. Any concerns about naming or structure before I submit a PR?
  3. Would you prefer harness code in ralph.sh or a separate script?

Happy to iterate based on your feedback.

Activity

1
Keesan124mo ago

The generator/evaluator split makes sense, especially if the goal is to stop one agent from grading its own homework.

The part I'd watch closely is where the hard boundary actually lives. In practice, the useful separation is less "builder vs judge" and more "mutator vs boundary".

A few failure modes show up fast in these loops:

  • the evaluator becomes another soft participant in the retry loop instead of a true gate
  • retries continue without a crisp stop reason
  • scores look informative but do not translate into a binary "advance / do not advance" decision

So the architecture feels strongest when the evaluator is explicitly fail-closed and produces evidence that is legible enough for a human to audit later, not just feedback for the generator in the moment.

If you ship this as an optional mode, I think the sharpest question is: what artifact proves why a story advanced, why it failed, and why it retried?

That answer will probably matter as much as the dual-agent split itself.