quarantine_stale_hnsw / quarantine_invalid_hnsw_metadata leave max_seq_id behind, so the rebuild they force is permanently incomplete (silent partial index; only filtered queries error)

Summary

Both quarantine paths in mempalace/backends/chroma.py rename a suspect HNSW segment directory but leave that segment's max_seq_id row in chroma.sqlite3. Chroma then treats the brand-new empty index as already synced through the old watermark and never replays anything below it.

The quarantine exists to force a rebuild. Instead it deterministically produces a permanently incomplete index — and collection.count() keeps reporting the full total, because that reads the metadata segment.

Still present on develop today:

• quarantine_stale_hnsw — rename at L541, no max_seq_id handling
• quarantine_invalid_hnsw_metadata — rename at L1655, same

Observed

Palace on mempalace 3.3.5 / chromadb 1.5.8, macOS. Three numbers that should agree:

Triggered by a .drift-20260831-094810 quarantine three days before I noticed. The palace holds 62 .drift-* directories spanning 2026-05-18 → 2026-08-31, the first dated 8 minutes after install — so this fires constantly, and each firing is a chance to truncate the index.

Why it stayed hidden — the part I think matters most

It fails asymmetrically. Unfiltered queries kept working and kept returning plausible, relevant results, because an unfiltered NN walk just traverses whatever nodes exist. Only where={"wing": ...} filtered queries raised:

Error executing plan: Internal error: Error finding id

because those resolve specific ids and look each one up in the index.

So for three days search silently answered from ~10% of the corpus while looking healthy. The first filtered query is what surfaced it. That is strictly worse than a hard failure, and I think it explains why this cluster keeps getting re-reported from the symptom end rather than the cause end (#1665, #1035, #1082, #1398, #2373 are all the filtered-query symptom).

Reproduction

1. Build a collection past hnsw:sync_threshold so a max_seq_id row is persisted.
2. mv <palace>/<vector-seg-uuid> <palace>/<vector-seg-uuid>.drift-TEST (exactly what quarantine does).
3. Reopen and run a filtered query.

Note step 1 — below the sync threshold no watermark row exists, so the bug does not reproduce. I could not reproduce the hard error synthetically at 1200 docs on 1.5.8; my evidence for the failure mode is the live palace above plus code inspection. Flagging that honestly rather than overclaiming.

Fix

Delete the row for the quarantined segment after a successful rename. The directory name is the segment UUID:

def _clear_segment_watermark(db_path: str, segment_id: str) -> None:
    try:
        with contextlib.closing(sqlite3.connect(db_path)) as conn:
            conn.execute("DELETE FROM max_seq_id WHERE segment_id = ?", (segment_id,))
            conn.commit()
    except sqlite3.Error:
        logger.exception(
            "Could not clear max_seq_id for quarantined segment %s — "
            "the index will rebuild INCOMPLETE.", segment_id,
        )

called after os.rename(...) in both quarantine functions. With no watermark, Chroma replays every record from embeddings and rebuilds a complete index.

This does not conflict with the _fix_blob_seq_ids decision

_fix_blob_seq_ids deliberately leaves max_seq_id alone because chromadb 1.5.x writes its own BLOB format there and misreading it via int.from_bytes(..., 'big') yields ~1.23e18 and suppresses writes (#1134, #1254).

DELETE never interprets the value. The concern that motivated "chromadb owns this column" is about rewriting it; removing a row for a segment whose index was just discarded is the one operation that is format-agnostic and provably safe.

repair --mode max-seq-id cannot catch this class

That detector looks for implausibly large watermarks left by the old shim (_MAX_PLAUSIBLE_SEQ_ID). 798 is entirely plausible — it is only wrong relative to an index that was just emptied. There is no absolute threshold that separates these; the check would have to compare the watermark against the index's actual element count.

Recovery for an already-broken palace

Clearing the watermark alone fixed unfiltered search but not filtered. Both steps were needed:

DELETE FROM max_seq_id WHERE segment_id IN ('<vector-seg-uuids>');  -- leave METADATA rows

python3 -m mempalace.repair rebuild --palace ~/.mempalace/palace --confirm

Verified afterwards in a fresh process (the MCP server caches segment handles and will mislead you in both directions): drawers unfiltered, wing= ×2, room= ×2, and closets — all OK across 981 + 28.

Minor, same area

Quarantine skips names containing .drift- or starting with ., so anything else with a dash and a data_level0.bin looks live. A backup I parked as <seg>.stale-watermark-<stamp> inside the palace got quarantined on the next scan. Widening the skip pattern, or documenting "never keep backups inside the palace directory", would help. Possibly related to the .drift- churn in #1715.

Related

#1710 (closest — quarantine destroying flushed vectors), #1526 (quarantine cascade + silent empty index), #1665 / #1035 / #1082 / #1398 / #2373 (the filtered-query symptom), #1963 (epic), #1715 (.drift- churn), #1134 / #1254 (the max_seq_id format hazard this proposal avoids).

Environment

mempalace 3.3.5 (dist-info 2026-05-18), chromadb 1.5.8, Python 3.9, macOS. Bug verified present on develop at time of filing.