#2443

repair: rebuild a single collection from SQLite when only mempalace_closets is wedged

OpenRemiAsselin42opened 1mo ago

What problem does this solve?

On 3.9.0 with chromadb 1.5.9 the closets HNSW segment stopped applying its log right after _purge_source_closets had run delete(where={"source_file": ...}) for 91 closets: the next closet upsert failed with "Error in compaction: Failed to apply logs to the hnsw segment writer", every later mine failed at its first closet write, and any client call on mempalace_closets, even count(), blocked forever (same shape as #1845, scoped to one collection while mempalace_drawers kept working). repair-status reported both collections OK because counts stayed within flush-lag tolerance, yet the VECTOR segment's max_seq_id sat 310 seq ids behind embeddings_queue. repair has no way to act on one collection: --segment only applies to --mode max-seq-id, and --mode from-sqlite re-embeds the whole palace (41k drawers here) to fix 1,525 closets.

What's the proposed solution?

A repair --collection <name> mode: read that collection's rows from chroma.sqlite3, rename or drop the wedged collection, recreate it and refile. That is what fixed it here by hand: SQL rename to mempalace_closets__broken, get_closets_collection(create=True), upsert of the 1,525 rows (about 20 min on CPU), then ChromaBackend.delete_collection on the renamed one, which completed even though count() on it never returned. A repair-status check comparing each VECTOR segment's max_seq_id with the queue would also have flagged the wedged segment while the counts looked fine.

Alternatives considered:

repair rebuild-index: full re-embed, hours on a large palace. Dropping the closets collection: loses the AAAK index layer for every already-mined file.

Activity

0

No activity yet.