#1558
Search-index backfill restarts from zero after a daemon restart (DELETE-first), leaving a large archive without keyword search for hours
On a 4.65M-message SQLite archive (309 GB), Store.BackfillFTS clears messages_fts with FTSClearSQL and then refills it in 5,000-id batches from MIN(id). Each batch commits, so the activity label's count is real progress, but nothing records where the build stopped: the next daemon start runs the DELETE again and begins at id 1.
What I saw (v0.21.0 then v0.22.0, macOS 27.2, Apple silicon, 192 GB):
- After a reboot the daemon reported
busy: building the search index (2170000/5820518 messages)2 h 26 m in. - I stopped the daemon to install 0.22.0 (an APFS clone taken at that moment holds 2,245,000
messages_ftsrows). - The 0.22.0 daemon, after its schema migration, started the build again. An hour later the live archive held 60,000
messages_ftsrows; the label read65000/5820518. - While the build runs,
msgvault searchprints "the search index is rebuilding or awaiting a rebuild in the background; results may be incomplete" and a keyword search over a 466,919-message Slack source returns nothing. The build also holds the operation gate as background work, so imports queue behind it. - Pace here: about 54,000 ids a minute with the Mac idle (a full pass ≈ 2 h), about 1,000 a minute while a test suite ran on the same machine.
Two asks:
- Make the backfill resumable. Keep the committed batches and continue from the highest indexed rowid (or from a stored cursor) when
NeedsFTSBackfillfinds the index incomplete, instead of deleting everything first. A gap-filling pass (INSERT ... WHERE id NOT IN (SELECT rowid FROM messages_fts)in ranges) would also repair an index that only misses the newest source's rows, which is the common case after a bulk import. - Let
rebuild-ftsstay the explicit "drop and redo" path, and have the daemon's automatic repair prefer the incremental path.
Also useful: a line in daemon status saying a restart will discard the build in progress, so operators do not restart into a redo.
Happy to test a build against this archive.
Activity
0No activity yet.