#1373

bug: block engine never writes again after recovering with 0 clean blocks

Opentonyalaribeopened 21d ago

Summary

A block engine that recovers with 0 clean blocks never writes to disk again. Every flusher waits for a clean block that nothing reclaims, and new entries are dropped once the submit queue fills. Reads of recovered entries still work, so the disk cache silently turns into a read-only snapshot.

Seen in production on foyer 0.22.6 (FS device):

foyer_storage::engine::block::recover: Recovers 157 blocks with data, 0 clean blocks, 138404 total entries ...

disk_write_bytes stayed at 0 after ~12.8 GB of inserts. The state is easy to reach: with clean_block_threshold = 1, a stop without close() between a flusher taking the last clean block and the follow-up reclaim leaves every block holding data.

Root cause

reclaim_if_needed is called only after taking a clean block, from on_writing_finish, and from on_reclaim_finish. With no clean block, get_clean_block parks a waiter without starting a reclaim, and init never reclaims. After recovering with 0 clean blocks, none of these triggers can fire.

Reproduction

  1. Fill a small block engine, then close().
  2. Overwrite every block file with a block that holds data, so recovery reports 0 clean blocks.
  3. Reopen, enqueue an entry, then wait(). It never returns.

Version

foyer 0.22.6; reproduces on main @ 0fb7628.

Activity

0

No activity yet.