Foyer not closing files, causing to exceed ulimit

Open

backend 84602 priyanshu 2503u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2504u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2505u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2506u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2507u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2508u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2509u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2510u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2513u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2514u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2515u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2516u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2517u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2518u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2519u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2520u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2521u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2522u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2523u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2524u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2525u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2526u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2527u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2528u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2529u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2530u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2531u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2532u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2533u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2534u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2535u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2536u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2537u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2538u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2539u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2540u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2541u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2542u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2543u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2544u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2545u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2546u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2547u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2548u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2549u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2551u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2552u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2553u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2554u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2555u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2556u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2557u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2558u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol
backend 84602 priyanshu 2559u      REG               1,18 1610612736            56369129 /Users/priyanshu/Aftershoot Projects/haskell_cache/kashe/smol

Above is the output for lsof -p $(pgrep backend) | tail -100, it has smol open like 2500 times.

The haskell_cache/smol is the cache unit, created using FileDeviceBuilder. I am running foyer 0.22.2.

Activity

MrCroxx8mo ago

Hi, Priyanshu. Thank you for reporting. Sorry for the late response. I missed the email.

Foyer will not proactively close cache files now, because each file may be accessed again. Frequently opening and closing them would cause significant performance overhead.

If the ulimit is too small, you can first consider adjusting the ulimit restriction. If having too many open files causes performance issues in your environment (for example, on some embedded devices—I’m not sure whether Foyer currently needs to serve in such environments), you can consider increasing the cache file size. If neither works, perhaps Foyer should consider supporting an open file timeout to proactively close files that haven’t been accessed recently.

Blonteractor8mo ago

Hey, thanks for replying. I understand foyer doesn't close open files as opening them again would be a performance drain. But it seems like its opening the same file multiple times. Even if I have only 20 blocks in my , those 20 blocks are opened a bunch of times and eventually exhaust the ulimit no matter what it is set to

MrCroxx8mo ago

Hey, thanks for replying. I understand foyer doesn't close open files as opening them again would be a performance drain. But it seems like its opening the same file multiple times. Even if I have only 20 blocks in my , those 20 blocks are opened a bunch of times and eventually exhaust the ulimit no matter what it is set to

Got it! Thank yoi for explaining. Let me check.

aquental1mo ago

I think I found why the same file shows up many times. On 0.22.6, a HybridCache never releases its device after close().await and drop. Every cache the process creates keeps its file handles until the tokio runtime shuts down. FileDevice opens the file once per cache (file.rs#L144, #L168), so an app that recreates its cache ends up with the same path open once per cache it ever created. That matches the lsof output above.

Repro (`FileDeviceBuilder`, same file every round)

[dependencies]
foyer = "=0.22.6"
tokio = { version = "1", features = ["full"] }

use std::time::Duration;

use foyer::{BlockEngineConfig, DeviceBuilder, FileDeviceBuilder, HybridCache, HybridCacheBuilder};

fn open_fds() -> usize {
    std::fs::read_dir("/dev/fd").unwrap().count()
}

async fn build(file: &std::path::Path) -> HybridCache<u64, Vec<u8>> {
    HybridCacheBuilder::new()
        .memory(16 * 1024 * 1024)
        .storage()
        .with_engine_config(
            BlockEngineConfig::new(
                FileDeviceBuilder::new(file)
                    .with_capacity(64 * 1024 * 1024)
                    .build()
                    .unwrap(),
            )
            .with_block_size(1024 * 1024),
        )
        .build()
        .await
        .unwrap()
}

fn main() {
    let dir = std::env::temp_dir().join(format!("foyer-file-leak-{}", std::process::id()));
    std::fs::create_dir_all(&dir).unwrap();
    let file = dir.join("smol");
    let rt = tokio::runtime::Runtime::new().unwrap();
    println!("baseline: fds {}", open_fds());

    rt.block_on(async {
        for round in 1..=5 {
            let cache = build(&file).await;
            cache.insert(round, vec![0u8; 4096]);
            cache.close().await.unwrap();
            drop(cache);
            tokio::time::sleep(Duration::from_millis(500)).await;
            println!("round {round}: closed+dropped  fds {}", open_fds());
        }
    });

    rt.shutdown_timeout(Duration::from_secs(5));
    println!("after runtime shutdown:  fds {}", open_fds());
    let _ = std::fs::remove_dir_all(&dir);
}

Output on macOS, with the stray ==========> lines removed (see the end of this comment):

baseline: fds 10
round 1: closed+dropped  fds 11
round 2: closed+dropped  fds 12
round 3: closed+dropped  fds 13
round 4: closed+dropped  fds 14
round 5: closed+dropped  fds 15
after runtime shutdown:  fds 6

lsof -p <pid> agrees: one more handle on smol after each round (1, 2, 3, 4, 5), and none after the runtime shuts down.

With FsDeviceBuilder it's worse, because that device opens one file per partition. A 64 MiB device with 64 KiB blocks leaks 1,024 descriptors per cache (baseline 13 → 1,037 → 2,061 → 3,085 over three rounds), all freed at runtime shutdown. We hit this through SlateDB's FoyerHybridCache, where a 4 GiB cache is 32,768 descriptors per handle and a few open/close cycles exceed macOS's per-process limit.

Cause

Links are for v0.22.6. The block engine's flusher task keeps itself alive through a reference cycle:

The flusher's Runner task (flusher.rs#L228) owns a BlockManager (flusher.rs#L312). Its loop exits only when its rx returns None (flusher.rs#L403).
The BlockManager holds the blocks, and therefore the partitions and the device file (manager.rs#L159). It also holds the reclaimer (manager.rs#L161).
The reclaimer is built with clones of the flushers (engine.rs#L366, reclaimer.rs#L54), and each clone holds the runner's tx (flusher.rs#L127).

So: runner → BlockManager → reclaimer → Flusher → the runner's own tx. The channel never closes, so the task never exits, and it keeps the device alive. close() (engine.rs#L561) only sets active = false and waits for pending flushes, so it doesn't break the cycle. Runtime shutdown drops the tasks, which is why the handles come back only then.

This means the files aren't just kept open on purpose while the cache is alive. They stay open after the cache is gone, so raising ulimit only delays the failure for any app that recreates caches.

Possible fixes

Any of these would break the cycle:

Have close() signal the runners to stop (a Submission::Close or a cancellation token), let them exit once idle, and await their join handles.
Have the reclaimer keep Weak references to the flushers or their senders.
Pass the flushers to the reclaimer per reclaim, instead of storing them behind the BlockManager.

Until a fix lands, a workaround is to give each cache its own tokio runtime (or process) and shut it down after close().

@Blonteractor, does your app create the HybridCache more than once in the same process, for example per request through the Haskell bindings? If so, this is likely the same bug.

Environment: foyer 0.22.6 (the close() path is unchanged on main), tokio 1.53.1, rustc 1.98.1, macOS 27 on Apple M3 Pro. Not tested on Linux or io_uring, but the cycle doesn't depend on the platform or I/O engine. I'm happy to test a patch.

Unrelated, but I noticed it while reproducing: FileDeviceBuilder::build has a leftover println!("==========> {capacity}") (file.rs#L103).

Foyer not closing files, causing to exceed ulimit

Activity

Repro (FileDeviceBuilder, same file every round)

Cause

Possible fixes

Repro (`FileDeviceBuilder`, same file every round)