#1380

fix(storage): spawn reclaim after releasing the block manager state lock

Openhuyz0wants to merge intofrom

BlockManager::reclaim_if_needed spawned the reclaim future while the caller held the state write lock. A spawner can drop the future immediately, e.g. tokio does so on the calling thread once its runtime is shutting down. Dropping the future drops its ReclaimingBlock, whose Drop calls on_reclaim_finish, which takes the same non-reentrant lock again and deadlocks the thread on itself.

reclaim_if_needed now returns the reclaim future, and get_clean_block, on_writing_finish and on_reclaim_finish spawn it after releasing the lock. reclaiming_blocks is still updated under the lock, so wait_reclaim and the reclaim concurrency limit behave as before.

Add test_reclaim_dropped_by_spawner_does_not_deadlock. It spawns reclaims onto a runtime that is already shut down, so each one is dropped immediately, and runs get_clean_block and on_writing_finish on a separate thread with a 10 s timeout.

Validation:

  • The new test fails with a timeout when the spawn is moved back under the lock, and passes with the fix.
  • The standalone reproduction in #1379 hung in 5 of 5 runs before and ran 500 iterations without hanging after.
  • cargo test -p foyer-storage: 29 lib tests passed.
  • cargo fmt --check and cargo clippy -p foyer-storage --all-targets are clean (apart from an existing load_throttle_switch dead-code warning).

#1374 also calls reclaim_if_needed from get_clean_block and init. Whichever lands second needs a small rebase: the new calls should spawn after dropping the lock, too.

Fixes #1379.