bug: cache entries that do not fit the tail of the flush buffer are silently and permanently dropped

• Component: foyer-storage, block engine flusher
• Severity: silent data loss — insert reports success and the entry never becomes readable from either the disk cache or the in-memory keeper

Summary

When a serialized entry does not fit the remaining space of the current flush buffer, the block engine's flush runner throws it away instead of writing it into the next buffer. It also drops the PieceRef, so PieceRef::drop (keeper.rs:134-149) removes the entry from the Keeper and it is gone from memory too. The only trace is an increment of storage_queue_buffer_overflow.

This is not a pathological-configuration edge case. It happens on the default configuration under any write burst, because the runner drains the submission channel into the current buffer before it considers rotating it, and there is no rotate-and-retry anywhere.

Where

Buffer::push returns bool, collapsing three distinct outcomes into false (foyer-storage/src/engine/block/buffer.rs):

The caller (flusher.rs:432-443) cannot tell them apart, so it treats all three as the last one and discards the piece. The BufferSizeLimit case is not even logged — the tracing::warn! at buffer.rs:215-217 is explicitly suppressed for it. Buffer::push_slice conflates the same two reasons in a single check (buffer.rs:270), so Submission::Reinsertion entries that the reinsertion filter explicitly picked to keep are discarded the same way.

Because transient rejections are counted the same as unstorable ones, storage_queue_buffer_overflow (metrics.rs:53) cannot distinguish them either, which is why the loss is invisible in practice.

Why it happens routinely

Runner::run (flusher.rs:347-407) drains everything available into the current buffer first and only then decides whether to rotate — gated on no_io_task. While an io task is in flight nothing can rotate, so every submission arriving in that window goes into the same buffer until it is full, and then every subsequent entry of the burst is dropped. There are exactly two io buffers (flusher.rs:194-195), so only one io task can be in flight at a time: this window is every flush, not a rare race.

Reproduction

BlockEngine::hold_flush blocks rotation, which is exactly the "io task in flight" condition without having to race it. With engine_for_test and buffer_pool_size = 16 KiB (io_buffer_size = 16 KiB, max_entry_size = block_size - blob_index_size = 12 KiB, and a 3 KiB entry occupies exactly one 4 KiB page):

// Four 3 KiB entries fill the 16 KiB buffer exactly.
store.hold_flush();
for i in 0..5 {
    enqueue(&store, memory.insert(i, vec![i as u8; 3 * KB]));
}
store.unhold_flush();
store.wait().await;

for i in 0..4 {
    assert!(store.load(memory.hash(&i)).await.unwrap().kv().is_some());
}
// Fails: entry 4 fits a fresh buffer and a block just fine, but it arrived when the tail of the current buffer was exhausted.
assert!(store.load(memory.hash(&4)).await.unwrap().kv().is_some());

A longer burst (e.g. 12 entries) loses everything after the first four, all of them ordinary entries well within max_entry_size.