Small objects can waste disk space and index memory when each object is stored as an individually aligned entry. This adds an independent, CacheLib BigHash-inspired engine selected through SmallEngineConfig.
The engine packs records into fixed-size buckets, locates buckets directly from key hashes, and uses an 8-byte Bloom filter per bucket instead of a per-object address index. Buckets evict records in FIFO order. Worker shards serialize mutations, bound queued work, and invalidate affected buckets on overflow so dropped replacements or deletes cannot expose stale values.
The change includes checksummed bucket encoding, compression, recovery modes, cancellation-safe clearing, shutdown error reporting, metrics, public exports, documentation, and a runnable restart/delete example. BlockEngine remains independent; automatic size-based routing is outside this change.
Tradeoffs: mutations read and rewrite a whole bucket, pending mutations temporarily hide that bucket, and overflow can evict neighboring entries. Recovery scans the device and requires stable capacity, bucket size, and key hashing. This is a disposable cache without fsync/WAL durability; performance has not been benchmarked.
Validation:
- Library tests for
foyer-storage,foyer, andfoyer-commonwithstrict_assertions,test_utils: 91 passed, 1 ignored. - Small-engine tests with
serde,tracing,strict_assertions,test_utils: 12 passed, including direct I/O, io_uring, recovery, overflow, corruption, concurrency, cancellation, and injected write failure. - Documentation tests: 4 passed.
cargo run --example small: passed.- Targeted all-target Clippy and
cargo fmt --all --check: passed; Clippy reports the existing unknownchunks_exact_to_as_chunkslint warning. - The full
cargo x --fastsuite and performance benchmarks have not been run.
This is a single-layer stack targeting main, with no prerequisite PRs.