Four months on, the #29 remediations have not shipped, plus six benchmark findings that audit did not cover

Summary

This is a follow-up to #29, which I found only after doing my own audit and which anticipated two of my findings. Credit to @dial481 for getting there first, and this issue is deliberately scoped to avoid restating that work.

Two things prompted filing anyway.

First, the remediations that closed #29 have not shipped. The closing comment on that issue committed to specific changes. Verified against develop at a9f345c (2026-08-28), four months later, none of the four are in the tree.

Second, I put a number on #29's central argument and found six further problems it did not cover. Rescoring the committed rankings under LongMemEval's own scorer gives 0.870 recall_all@5, not 0.966. The remaining findings are in the abstention filter, the NDCG normalisation, the session relabelling, the held-out split, ConvoMem's category discovery and MemBench's hit criterion.

I also believe open issue #242 is incorrect, and that acting on it as planned would move the harness away from the official protocol rather than towards it. That is section 4.

1. What #29 closed on, and what is in the tree today

From the closing comment on #29:

benchmarks/BENCHMARKS.md has not been modified since c35686c (2026-05-08), and that commit was about recommending uv.

Inside the harness, recall_all is computed seven times and printed zero times. The value needed to honour the first commitment is already in the dict at longmemeval_bench.py:78; every consumer takes the any key instead (:3179-3181, :3209, :3226, :3232).

I am not reading motive into this. Four months of feature work on a repository this active is more than enough for a docs commitment to be genuinely forgotten. But #29 is closed as completed, which means the public record says these were fixed, and they were not.

2. Putting a number on #29's recall_any finding

#29 established the substitution qualitatively: recall_any gives full credit for finding one of N evidence sessions, and "roughly multiplies the odds of full credit by the number of gold sessions per question." It did not rescore.

Rescoring the committed rankings in benchmarks/results_mempal_raw_session_20260414_1629.jsonl under src/evaluation/print_retrieval_metrics.py, which selects recall_all@5 and drops the _abs questions:

No retrieval is re-run. These are the same ranked lists, scored the way the benchmark specifies. A standard-library script that reproduces it is at the bottom.

The evidence-count distribution is what drives the gap. In longmemeval_s_cleaned.json:

324 of 500 questions are scored as solved on the strength of one session out of up to six, and the damage is concentrated exactly where the docs claim the most strength:

The three single-session categories do not move, which is the confirmation: they are the ones with a single evidence session.

3. Six findings not covered by #29, #39, #214, #242, #314 or #367

I searched the issue tracker for each of these before filing.

3.1 The 30 abstention questions are scored as retrieval

print_retrieval_metrics.py drops them on its first line:

in_data = [x for x in in_data if '_abs' not in x['question_id']]

This harness keeps all 30, and they score 0.967 R@5, marginally above the overall mean. These are the questions whose correct behaviour is to abstain. Scoring them as retrieval measures the opposite of what they exist to test, and lifts the average rather than probing anything.

3.2 NDCG normalises against the retrieved set, not the corpus

longmemeval_bench.py:61-68:

relevances = [1.0 if corpus_ids[idx] in correct_ids else 0.0 for idx in rankings[:k]]
ideal = sorted(relevances, reverse=True)
idcg = dcg(ideal, k)

The ideal is built from the top-k that was retrieved. LongMemEval's eval_utils.py::ndcg builds it from the relevance vector over the whole corpus:

relevances = [1 if doc_id in correct_docs else 0 for doc_id in corpus_ids]
ideal_relevance = sorted(relevances, reverse=True)

So evidence you never retrieved cannot lower your NDCG, because IDCG behaves as though it never existed. A question with three evidence sessions where you retrieve one, at rank 1, scores 1.0. Corrected, raw-mode NDCG@10 is 0.783 rather than the documented 0.889.

3.3 The official noans relabelling is not applied

run_retrieval.py:209 relabels an evidence session to noans when none of its user turns carries has_answer, because at session granularity the indexed text is user turns only, so such a session holds no retrievable evidence:

if 'answer' in sess_id and all([not turn['has_answer'] for turn in [x for x in data if x['role'] == 'user']]):
    ids = [sess_id.replace('answer', 'noans')]

longmemeval_bench.py:3078 uses entry["answer_session_ids"] verbatim. This changes the ground truth for 113 of 500 questions. My table in section 2 applies the official relabelling; without it, recall_all@5 is 0.857 instead of 0.870. Either way the conclusion holds, but the two should not be mixed when quoting a figure.

3.4 The held-out 450 contains all three questions hybrid_v4 was tuned on

#29 made the general point that the split was carved out after the patches. Here is the specific consequence, which I have not seen stated: the three questions are not merely somewhere in the 500, they are all three in the held-out 450, and they are exactly the questions the patches moved.

From benchmarks/lme_split_50_450.json, all of d6233ab6, 4dfccbf8 and ceb54acb resolve to held_out. Evidence-session rank in the committed result files:

Three misses converted to three hits, inside the set presented as "never tuned on", worth 0.67pp of it. Under the official metric that run is 90.1% recall_all@5, not 98.4%.

3.5 ConvoMem runs 5 of the 6 categories, and the missing one is the multi-evidence one

convomem_bench.py:88 hardcodes the 1_evidence subdirectory during discovery. On HuggingFace, changing_evidence publishes no 1_evidence directory, only 2_evidence through 6_evidence. Discovery returns an empty list, SAMPLE_FILES["changing_evidence"] is None, and the category is skipped with a one-line message that is easy to miss in a long run.

benchmarks/results_convomem_raw_top10_20260414_1649.json confirms it: 250 items, 5 categories, no changing_evidence. benchmarks/README.md presents this run as "Run all categories, 50 items each (our 92.9% result)"; BENCHMARKS.md describes ConvoMem as "six categories of conversational memory". The omitted category is the only one where recall = found / len(evidence_texts) would have exercised the multi-evidence path.

Two smaller points on the same harness:

• Every item under 1_evidence is 1 conversation of roughly 42 messages with 1 evidence message, retrieved at top-10. ConvoMem's stated purpose is scaling to 5, 10, 50, 100 and 300 conversations per item, so the slice being measured is the smallest available.
• The abstention_evidence category is scored by whether a related but non-answering message was retrieved, so "Abstention 91.0% (Strong)" in the per-category table is not measuring abstention.

I specifically checked the loose substring matcher at :207-211, expecting false positives, and it is fine. ConvoMem stores evidence as a trimmed span of the containing message, so ev_text in ret_text is doing necessary work. No change needed there.

3.6 MemBench reports any-target hit, including on questions with more targets than k

membench_bench.py:384:

hit = bool(target_sids & set(retrieved_sids)) or bool(target_sids & set(retrieved_global))

From the committed results_membench_hybrid_all_movie_top5_20260414_1656.json, 8,500 questions:

6,244 of 8,500 questions have more than one target. 1,000 of them (11.8%) have 8 or 10 targets, so complete recall at k=5 is arithmetically impossible, and every one of those can still be scored a hit.

4. #242 is incorrect, and the planned fix would deviate from the official protocol

The closing comment on #29 lists "Fixing #242 (@bobmatnyc's assistant-turn discard bug in the bench harness) before re-running any public benchmark." I think that would be a mistake, for two reasons.

The official runner discards assistant turns too. run_retrieval.py:206, session granularity:

text = ' '.join([interact['content'] for interact in data if interact['role'] == 'user'])

That is the same construction #242 calls a bug. Indexing user turns only is the LongMemEval session-level protocol, not a MemPalace deviation from it. Adopting #242's patch would make the harness's numbers incomparable to any published LongMemEval session-level retrieval result, which is the opposite of what #29 was asking for.

#242's factual claim about this repository's result does not hold. It states that "MemPalace's published 96.6% headline result was measured on a 100-question sample that happened to contain zero single-session-assistant entries, so the gap never surfaced." The committed 500-question file contains 56 single-session-assistant entries and they score 0.964 R@5, the joint second-best of the six categories, not 0%. The issue title says 0% and its own impact table says 62.5%, measured on kuzu-memory rather than on this harness.

There is a real question underneath it, which is whether session-level retrieval on user turns is a good way to evaluate assistant-attributed questions. That is a critique of LongMemEval's design, and it belongs upstream, the way @dial481 pointed out in #314. It is not a bug in this repository.

What I checked and found sound

For balance, since the rest of this issue is critical:

1. No label leakage. entry["answer"] appears once in 3,441 lines, on the logging line at :3199. It never reaches the retriever.
2. The committed results are authentic. I rebuilt the session documents from longmemeval_s_cleaned.json and compared them to the text logged in the results file: 600 of 600 exact matches, including the "\n".join(user turns) construction. That file is the output of this code on this data.
3. Corpus construction matches the official protocol, as covered in section 4.
4. The headline reproduces internally. Recomputed from the committed log: 0.966 / 0.982 / 0.889, exactly as documented.
5. hybrid_v4's three fixes are generic code, not question ids in a lookup table.

The underlying result is genuinely interesting. 87.0% recall_all@5 with an off-the-shelf 384-dimensional embedder and no LLM anywhere is a strong argument for the verbatim-storage thesis, and it is the version that survives contact with the official scorer.

Reproduction

No chromadb, no re-running retrieval, standard library only.

curl -fsSL -o longmemeval_s_cleaned.json \
  https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json

python rescore_official.py longmemeval_s_cleaned.json \
  benchmarks/results_mempal_raw_session_20260414_1629.jsonl

reported recall_any@5, all 500 questions : 0.966
official recall_all@5, 470 non-abs       : 0.870
official recall_all@10                   : 0.945
official ndcg_any@10                     : 0.783

#!/usr/bin/env python3
"""
Rescore a MemPalace LongMemEval result file under the official LongMemEval
metric, as implemented in LongMemEval's own
src/evaluation/print_retrieval_metrics.py and src/retrieval/eval_utils.py.

Three differences from benchmarks/longmemeval_bench.py:
  1. the official scorer reports recall_all@k, not recall_any@k
  2. the official scorer drops the 30 '_abs' (abstention) questions
  3. the official NDCG ideal is built over the corpus, not the retrieved top-k

Needs no chromadb and re-runs no retrieval: it reads the rankings that are
already committed in benchmarks/results_*.jsonl.

Usage:
    python rescore_official.py longmemeval_s_cleaned.json benchmarks/results_....jsonl
"""
import json
import math
import sys


def stream_json_array(path, chunk=1 << 22):
    """Iterate a top-level JSON array without holding the 277MB file in memory."""
    dec, buf, started = json.JSONDecoder(), "", False
    with open(path, encoding="utf-8") as f:
        while True:
            data = f.read(chunk)
            if not data:
                return
            buf += data
            i = 0
            while True:
                while i < len(buf) and buf[i] in " \t\r\n,":
                    i += 1
                if i < len(buf) and not started and buf[i] == "[":
                    started, i = True, i + 1
                    continue
                if i >= len(buf) or buf[i] == "]":
                    break
                try:
                    obj, i = dec.raw_decode(buf, i)
                except ValueError:
                    break
                yield obj
            buf = buf[i:]


def dcg(relevances, k):
    """LongMemEval src/retrieval/eval_utils.py::dcg."""
    rel = relevances[:k]
    if not rel:
        return 0.0
    return rel[0] + sum(r / math.log2(j + 2) for j, r in enumerate(rel[1:]))


def ndcg_official(ranked_relevances, n_correct, n_corpus, k):
    """Ideal is built over the WHOLE corpus, not over the retrieved top-k."""
    ideal = [1.0] * min(n_correct, n_corpus) + [0.0] * max(0, n_corpus - n_correct)
    idcg = dcg(ideal, k)
    return 0.0 if idcg == 0 else dcg(ranked_relevances, k) / idcg


def official_labels(entry):
    """
    LongMemEval run_retrieval.py::process_item_flat_index relabels an evidence
    session to 'noans' when no USER turn in it carries has_answer, because at
    session granularity the indexed text is user turns only. correct_docs is
    then every corpus id containing the substring 'answer'.
    """
    labels = []
    for sid, session in zip(entry["haystack_session_ids"], entry["haystack_sessions"]):
        if "answer" not in sid:
            continue
        if any(t.get("has_answer") for t in session if t["role"] == "user"):
            labels.append(sid)
    return set(labels)


def main(data_file, result_file):
    labels = {e["question_id"]: official_labels(e) for e in stream_json_array(data_file)}
    rows = [json.loads(line) for line in open(result_file, encoding="utf-8")]

    reported, strict5, strict10, ndcg10 = [], [], [], []
    for r in rows:
        qid = r["question_id"]
        ranked = [x["corpus_id"] for x in r["retrieval_results"]["ranked_items"]]
        reported.append(r["retrieval_results"]["metrics"]["session"]["recall_any@5"])
        if qid.endswith("_abs"):          # the official scorer drops these
            continue
        correct = labels[qid]
        strict5.append(float(correct <= set(ranked[:5])))
        strict10.append(float(correct <= set(ranked[:10])))
        rel = [1.0 if c in correct else 0.0 for c in ranked]
        ndcg10.append(ndcg_official(rel, len(correct), 50, 10))

    def mean(xs):
        return sum(xs) / len(xs)

    print(f"file: {result_file}")
    print(f"  reported recall_any@5, all {len(reported)} questions : {mean(reported):.3f}")
    print(f"  official recall_all@5, {len(strict5)} non-abs        : {mean(strict5):.3f}")
    print(f"  official recall_all@10                              : {mean(strict10):.3f}")
    print(f"  official ndcg_any@10                                : {mean(ndcg10):.3f}")


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Applied to the other committed files: results_mempal_hybrid_v4_held_out_session_20260414_1634.jsonl gives 0.901 recall_all@5 against a reported 0.984, and results_mempal_hybrid_v4_llmrerank_session_20260414_1659.jsonl gives 0.919 against a reported 0.992.

Suggested changes

Ordered by how much they matter.

1. Honour the first #29 commitment, or reopen #29 and say it is not happening. Either is fine; the current state, where it is closed as completed and the headline is unchanged on four surfaces, is the one option that misleads.
2. Print recall_all@k as Recall@k and drop _abs from the aggregate. recall_all is already computed at :78. This is a change to which key is printed, plus a filter. Keep recall_any as a clearly labelled secondary line; it is informative, just not the benchmark's metric.
3. Fix the NDCG ideal to be built over the corpus relevance vector.
4. Apply the official noans relabelling so a session with no user-turn evidence is not a session-level target.
5. Close #242 as not-a-bug, citing run_retrieval.py:206, and do not gate the re-run on it. If assistant-turn indexing is wanted, add it as a separate labelled mode so the official-protocol number stays available.
6. Retire the 98.4% held-out figure, or re-derive it from a split created before any tuning.
7. Make ConvoMem's category discovery fail loudly rather than skipping, and label the 92.9% as 5 categories, 1_evidence, 1 conversation per item.
8. Report MemBench as all-targets hit@5, or state that it is any-target and that 11.8% of questions cannot be satisfied completely at k=5.
9. Apply #875's caveat to the ConvoMem and LoCoMo tables at BENCHMARKS.md:104-109, or remove them, consistently with what docs/HISTORY.md records as done.

One piece of context for LoCoMo that needs a sentence rather than a code change: with 19 to 32 sessions per conversation and k=10, a random ranker scores 0.371. The 60.3% and 88.9% figures are real signal above that, but the pool size belongs next to them, the way the top-50 caveat already is.

Why file this

docs/HISTORY.md and the integrity section of BENCHMARKS.md are more candid than most projects manage, and #29 got a genuinely good response. That is the reason for filing rather than a reason not to: the gap here is not between what the project knows and what it admits, it is between what it admitted and what it shipped. The corrected number is good enough to publish on its own, which makes the gap unnecessary.