virtiofs root: small-file metadata 10-30x slower than tmpfs and writes are write-through (1 request queue, 1 worker thread, writeback=false)

Summary

With a virtiofs root (crun-krun / krun_add_virtiofs2(ctx, "/dev/root", "/", ...)), file-metadata operations in the guest are 10-30x slower than in-guest tmpfs, and writes are write-through (the same speed with and without fsync). From the code, the virtiofs device has one request queue, one worker thread and writeback: false with no way to change any of them through the C API. We hit this running CI builds (Android/Gradle) on SIPB's Hyades runner; builds are dominated by small-file work, so the root filesystem is the bottleneck. Related reports below (#655, #667, #414) point the same way. I am asking whether this is intended and what direction would be accepted; I can test patches on this setup.

Measurements and write-up were prepared with an AI coding assistant (Claude Code) from the CI job logs; I reviewed the numbers.

Setup

1. Forgejo Runner + rootless Podman + crun with the krun handler (engine.runtime = "krun"); the SIPB forks of libkrun ("2.0-sipb", rev bd369ebb4fb2) and crun do not touch the fs code as far as I can tell. Config: https://forgejo.mit.edu/SIPB/hyades (see overlays.nix, sipbphus/default.nix).
2. Guest: libkrunfw kernel 6.12.91, 16 vCPU (AMD EPYC, KVM), 62.8 GiB RAM (krun.ram_mib=65536), job runs as root.
3. /proc/self/mountinfo: virtiofs /dev/root / rw,relatime (no dax), although the kernel log reports virtiofs virtio3: Cache len: 0x20000000 (a 512 MiB DAX window exists). rootfstype=virtiofs rw quiet no-kvmapf init=/init.krun. /dev/shm is a guest tmpfs.
4. I have not re-run this outside the Hyades runner, so I cannot say how much depends on the host's backing store. The numbers below come from one job; a 2-pass repeat varied by under 5 percent.

Measurements (same script on each filesystem, 5000 files of 4 KiB, 1 GiB sequential)

1. Write throughput does not depend on fsync on virtiofs (189 vs 178 MiB/s): consistent with write-through, no guest write coalescing.
2. fio on virtiofs (buffered, psync, 4 jobs): random 4k read 51,946 IOPS (75 us), random 4k write 50,426 IOPS (78 us), 4k write plus fsync 1,063 IOPS (148 us). Cold random 4k reads through the loop device are worse (10,169 IOPS), so a loop image is no free lunch for read-heavy cold data.
3. 16 processes in parallel on virtiofs: 108,751 stat/s (about 8x one process), 9,916 creates/s: latency-bound rather than saturated.
4. Realistic: unzip of a 137 MB zip 2.79 s vs 1.31 s on loop-ext4; extracting a 120 MB repo tarball 3.59 s vs 0.29 s; git status 0.84 s vs 0.43 s; rm -rf 0.75 s vs 0.26 s.

Repro (run inside any krun container; compare /tmp on virtiofs with /dev/shm):

import os, sys, time
d = sys.argv[1]; os.makedirs(d, exist_ok=True); n = 5000; body = b"x" * 4096
ps = [f"{d}/f{i}" for i in range(n)]
def t(f):
    a = time.perf_counter(); f(); return n / (time.perf_counter() - a)
def create():
    for p in ps: open(p, "wb").write(body)
print("create/s", int(t(create)))
print("stat/s", int(t(lambda: [os.stat(p) for p in ps])))
print("open+read/s", int(t(lambda: [open(p, "rb").read() for p in ps])))
print("unlink/s", int(t(lambda: [os.unlink(p) for p in ps])))
buf = os.urandom(1 << 20)
for sync in (False, True):
    a = time.perf_counter()
    with open(f"{d}/big", "wb") as f:
        for _ in range(1024): f.write(buf)
        f.flush()
        if sync: os.fsync(f.fileno())
    print("write MiB/s fsync=%s" % sync, int(1024 / (time.perf_counter() - a))); os.unlink(f"{d}/big")

What I found in the code (main, v1.19.6)

1. src/devices/src/virtio/fs/device.rs: config.num_request_queues = 1; and the passthrough::Config is built with ..Default::default() (only root_dir, semantics and attr_timeout are set).
2. src/devices/src/virtio/fs/linux/passthrough.rs: the defaults are cache_policy: Default::default() (Auto) and writeback: false; the field documentation says writeback "can improve performance as it allows the FUSE client to cache and coalesce multiple writes" when the host has exclusive access to the directory.
3. src/devices/src/virtio/fs/worker.rs: FsWorker::run spawns a single thread that serves all queues.
4. No setter for any of these in the libkrun C API (krun_add_virtiofs2..4 take tag, path, DAX window size and permission semantics). I may well have missed something; please correct me.

For a private, disposable root filesystem (every CI job's container rootfs), the exclusive-access condition for writeback holds, so this looks like a reasonable thing to offer as an option.

Questions

1. Are one queue, one worker thread and writeback: false intentional defaults for the Linux passthrough backend, or just where it started?
2. Would you accept an API to set writeback and cache_policy for a virtiofs device (for example through an extension of krun_add_virtiofs4), and, separately, a multi-queue or worker-pool server? I can prototype the first and benchmark on this setup.
3. For CI-style workloads (many small files), is a virtio-blk root the recommended answer? #655 suggests it is (maintainer comment, 2026-05-04), but crun has no annotation to attach a disk outside the SEV variant (see #667 item 4).

Related

1. #655 (closed): make -j16 11.22 s on virtiofs root vs 6.05 s on libvirt; same speed with virtio-blk root.
2. #667 (open): CI runner use case, asks for a block device key in .krun_vm.json.
3. #414 (open): podman build performance versus vfkit, attributed to virtio-fs mounts.
4. #824 (open): virtio-blk services one request at a time per disk.
5. #515 and containers/crun#1952: --tmpfs under crun-krun is host-side and goes through virtiofs; only /dev/shm is a guest tmpfs.
6. SIPB's Hyades README TODO lists "virtiofs performance tweaks (or just use tmpfs?)".

Update (2026-10-14)

1. Same commit, one run each of a real Android/Gradle CI job (JDK and SDK install, compile, four targeted test steps): 458 s with the root on virtiofs, 393 s with the workspace, Gradle home and SDK on a guest tmpfs, 324 s on an ext4 image on a loop device. Compile alone: 276 s, 203 s, 183 s. The jobs share a host, so treat differences under about 20% as noise.
2. The build used 2.6 GiB of disk and up to 11.7 GiB of RAM, so a tmpfs is enough for it.
3. The numbers and suggestions for the runner's operators are in https://forgejo.mit.edu/SIPB/hyades/issues/2.
4. Unrelated to the filesystem but found on the same runner: a TSI hang in shutdown() while another thread is blocked in accept()/read(): #926.