Background
Ralph is an autonomous AI agent loop that implements features story-by-story, maintaining context across iterations via progress.txt. For long-running tasks this file grows unboundedly — every new story must load the entire history even when only a small slice is relevant.
Problem
progress.txt is append-only and flat. On each new user story, Ralph loads the full file to get context. On large projects this means loading thousands of lines of history to find a handful of relevant patterns, wasting tokens and increasing latency.
Proposed Solution: Indexed Progress Table
Maintain a lightweight index table at the top of progress.txt that maps topics to small reference files stored in a ralph/references/ folder.
Index format (top of progress.txt)
## Progress Index
| Topic | Reference Path |
|---|---|
| Auth flow learnings | ralph/references/auth-flow.md |
| DB schema decisions | ralph/references/db-schema.md |
| API error handling patterns | ralph/references/api-errors.md |Rules for the agent
When completing a story:
- If a reusable pattern or learning emerged, write it to
ralph/references/<topic>.md - Add or update its row in the index table at the top of
progress.txt - Append the normal one-line narrative summary to the log as usual
When starting a story:
- Read only the index table (always tiny — a few KB regardless of project size)
- Identify which reference files are relevant to this story's acceptance criteria
- Read only those files — skip the rest
CLAUDE.md addition (minimal)
## Context Indexing
Maintain a progress index table at the top of progress.txt:
- Column 1: Topic (short human-readable name)
- Column 2: Path to ralph/references/<file>.md
When completing a story:
1. If a reusable pattern/learning emerged, write it to ralph/references/<topic>.md
2. Add or update its row in the index table
3. Append a one-line summary to the narrative log as usual
When starting a story:
1. Read ONLY the index table
2. Identify which reference files are relevant to this story's acceptance criteria
3. Read only those files — skip the restWhy This Works
| Property | Today | With Index |
|---|---|---|
| Context load per story | O(total history) | O(number of topics) |
| Relevant signal | Buried in full log | Fetched by name |
| Backward compatible | — | Yes — projects without ralph/references/ behave exactly as today |
| Agent judgment needed | Must scan everything | Scan table → read 1–2 files |
The index stays small forever regardless of project length. Agents already have the judgment to decide what's "relevant" — giving them a structured table makes that judgment cheap (scan 10 rows, read one file) instead of expensive (load the full history).
This also complements the existing CLAUDE.md-per-directory pattern: directory-level CLAUDE.md files capture spatial knowledge (what lives where), while the references/ index captures temporal knowledge (what was learned across stories).
Relation to Existing Work
This is in the spirit of the token-efficiency RFCs already open (the Harness Mode discussions) but targets a different layer: the persistent context store rather than the agent loop architecture. The two are complementary and independent.
Implementation
The change is purely additive — a small addition to CLAUDE.md and an optional ralph/references/ folder. Projects that don't adopt it behave exactly as today.
Happy to open a PR if there's interest.
Interesting direction. The main thing this solves well is not just token volume, but retrieval ambiguity.
In long-running agent loops we've seen the real failure mode become: the run keeps rereading its own narrative log because that's the only durable memory it has, so every retry gets more expensive and less selective at the same time.
The split that has helped in practice is:
That keeps the agent from treating history as if all of it deserves to be loaded again.
One thing I would add to the RFC: a compaction / aging rule for the references layer itself. Otherwise
progress.txtgets smaller, butreferences/slowly becomes the new graveyard. Even a light rule like "promote only decisions that change future implementation behavior" would keep the index high-signal.Feels like a good fit for long-running tasks where context bloat becomes its own tax.