#1558

Search-index backfill restarts from zero after a daemon restart (DELETE-first), leaving a large archive without keyword search for hours

OpenJoiopened 2d ago

On a 4.65M-message SQLite archive (309 GB), Store.BackfillFTS clears messages_fts with FTSClearSQL and then refills it in 5,000-id batches from MIN(id). Each batch commits, so the activity label's count is real progress, but nothing records where the build stopped: the next daemon start runs the DELETE again and begins at id 1.

What I saw (v0.21.0 then v0.22.0, macOS 27.2, Apple silicon, 192 GB):

  • After a reboot the daemon reported busy: building the search index (2170000/5820518 messages) 2 h 26 m in.
  • I stopped the daemon to install 0.22.0 (an APFS clone taken at that moment holds 2,245,000 messages_fts rows).
  • The 0.22.0 daemon, after its schema migration, started the build again. An hour later the live archive held 60,000 messages_fts rows; the label read 65000/5820518.
  • While the build runs, msgvault search prints "the search index is rebuilding or awaiting a rebuild in the background; results may be incomplete" and a keyword search over a 466,919-message Slack source returns nothing. The build also holds the operation gate as background work, so imports queue behind it.
  • Pace here: about 54,000 ids a minute with the Mac idle (a full pass ≈ 2 h), about 1,000 a minute while a test suite ran on the same machine.

Two asks:

  1. Make the backfill resumable. Keep the committed batches and continue from the highest indexed rowid (or from a stored cursor) when NeedsFTSBackfill finds the index incomplete, instead of deleting everything first. A gap-filling pass (INSERT ... WHERE id NOT IN (SELECT rowid FROM messages_fts) in ranges) would also repair an index that only misses the newest source's rows, which is the common case after a bulk import.
  2. Let rebuild-fts stay the explicit "drop and redo" path, and have the daemon's automatic repair prefer the incremental path.

Also useful: a line in daemon status saying a restart will discard the build in progress, so operators do not restart into a redo.

Happy to test a build against this archive.

Activity

0

No activity yet.