quarantine_stale_hnsw destroys flushed vectors when the WAL has been purged (sibling path to #1710)

Open

What happened?

A 347k-drawer palace silently lost 47,099 embeddings — its entire first month, contiguous — with no error. Search kept working over the surviving tail, so the loss stayed invisible until a wing that should have had hundreds of hits returned almost none.

quarantine_stale_hnsw renamed a segment aside and let Chroma rebuild from the WAL, but chromadb 1.5.x purges embeddings_queue by default, so the rebuild replayed only from the purge watermark (seq 51,396 here). Everything below it was gone, and the rename counted as success. From hook.log:

2026-07-08 15:39:36  Quarantined corrupt HNSW segment … (sqlite 694s newer than HNSW and integrity check failed); renamed to …c33345ec….drift-20260708-153936

No SQLite metadata was lost (0 orphans), so repair --yes recovered all 347k drawers in ~3.5h. Nothing is permanently gone if the .drift-* dir survives. The defect is the silence: meanwhile filtered queries errored with Error finding id while unfiltered KNN returned confident results over two thirds of the corpus.

Same failure class as #1710, different guard. #1710 closed for quarantine_invalid_hnsw_metadata; quarantine_stale_hnsw (mempalace/backends/chroma.py:455) still quarantines against a WAL it never checks.

What did you expect?

Quarantine shouldn't be able to lose flushed vectors. Before renaming, compare the embeddings_queue floor against the collection's embedding count; if the WAL can't rebuild the collection, fail loudly and require an explicit repair instead of renaming.

How to reproduce:

Grow a palace until chromadb purges embeddings_queue behind the current segment.
Get data_level0.bin's mtime >300s behind chroma.sqlite3 and fail the index_metadata.pickle sniff test (continuous mining plus an interrupted shutdown).
Open the palace. The segment is renamed *.drift-<stamp> and the rebuilt index starts at the purge watermark. No error; mempalace status still shows the full SQLite count.

Environment:

OS: macOS 26 (Darwin 25.6.0), Apple Silicon
Python version: 3.14
MemPal version: MemPalace 3.9.0, chromadb 1.5.9

Investigation

Why the loss is contiguous. After the rename, the rebuild replays embeddings_queue from its current floor, and everything below had already been purged — so the gap is one solid block at the bottom rather than scattered holes. mempalace never sets automatically_purge itself (no occurrences in the 3.9.0 tree), which is the point: the guard depends on a WAL whose retention it neither controls nor checks.

3.9.0 narrowed the trigger but not the damage. The current guard has a stage-2 gate the version that ate these vectors lacked:

if not payload_corrupt and _segment_appears_healthy(seg_dir):
    logger.info(... "flush-lag, not corruption. Leaving in place.")
    continue

Real improvement, but os.rename(seg_dir, target) at line 541 still proceeds with no check that the WAL can reconstruct what it replaces. A segment that genuinely fails _segment_appears_healthy on a purged-queue palace still loses everything below the watermark.

Where to start. _vector_segment_id already reads chroma.sqlite3 directly to avoid opening a segfault-prone segment, so the access pattern for the pre-rename check exists in this module. The >= relaxation from #1723 is present in both 3.8.0 and 3.9.0 (chroma.py:1557, identical); the remaining #1710 fix hasn't shipped (3.9.0 published 2026-08-31, two weeks before the close).

Full timeline, both hook.log firings and the 0-orphan verdict: my 2026-08-31 comment on #1710.

Activity

No activity yet.