RFC: Refine the design of Device and related interfaces for various cache engine support

Summary

This RFC proposes refining the design of the Device trait in Foyer to better support multiple cache engines and heterogeneous storage media.

Instead of requiring each storage media to support all cache engines, we propose:

• Making Device expose only generic, media-agnostic capabilities (e.g., statistics, throttle, capacity).
• Moving engine-specific read/write semantics into separate, engine-specific traits.
• Letting engines depend only on the capabilities they actually need.

This avoids compatibility explosion between engines and media.

Motivation

Foyer aims to support multiple disk cache engines:

• Block-based cache engine (already implemented)
• Object-based cache engine
• Small-entry disk cache engine

And multiple storage media:

• File system directory
• File (raw or regular file)
• OpenDAL (planned)

The current Device abstraction is designed primarily around block-based IO. This makes it less suitable for:

• Object-based engines
• Object-oriented media such as OpenDAL

Requiring every media to support every engine introduces unnecessary complexity and tight coupling.

Problem

Today, Device mixes:

• Generic device information (statistics, throttle)
• Specific IO semantics (block read/write)

However, not all engines require block IO, and not all media can naturally provide it.

If we continue this design, we risk:

• Engine × Media compatibility explosion
• Artificial emulation layers (e.g., object store pretending to be block device)
• Increased maintenance cost

Proposal

1. Refine Device to a minimal, generic trait

Device should expose only common, media-agnostic capabilities:

pub trait Device: Send + Sync {
    /// Get the capacity of the device.
    ///
    /// NOTE: `capacity` must be 4K aligned.
    fn capacity(&self) -> usize;

    /// Get the allocated space in the device.
    fn allocated(&self) -> usize;

    /// Get the free space in the device.
    fn free(&self) -> usize {
        self.capacity() - self.allocated()
    }

    /// Get the statistics of the device this partition belongs to.
    fn statistics(&self) -> &Arc<Statistics>;
}

It should not define read/write methods.

2. Introduce engine-specific capability traits

Each engine defines the IO capability it requires.

Example: Block engine

pub trait BlockEngineDevice: Device {
    /// Create a new partition with the given size.
    ///
    /// NOTE:
    ///
    /// - Allocating partition may consume more space than requested.
    /// - `size` must be 4K aligned.
    fn create_partition(&self, size: usize) -> Result<Arc<dyn Partition>>;

    /// Get the number of partitions in the device.
    fn partitions(&self) -> usize;

    /// Get the partition with given id in the device.
    fn partition(&self, id: PartitionId) -> Arc<dyn Partition>;
}

(Partition-related APIs are omitted here)

Example: Object engine

pub trait ObjectEngineDevice: Device {
    async fn get(&self, key: &StorageKey) -> Result<Bytes>;
    async fn put(&self, key: &StorageKey, value: StorageValue) -> Result<()>;
    async fn delete(&self, key: &StorageKey) -> Result<()>;
}

Engines depend only on the trait they need:

impl<D: BlockEngineDevice> BlockCacheEngine<D> {
    // ...
}

Benefits

• Avoids forcing every media to support every engine.
• Keeps abstraction clean and capability-based.
• Makes OpenDAL integration natural (via ObjectDevice).
• Improves extensibility for future engines.