Under normal circumstances, virtio devices operate in an asynchronous way. While in the guest, the vCPU enqueues the request in the virtqueue, kicks it (which generates a brief vm-exit) and moves to other tasks. Once the VMM has processed the request and added a reply to the virtqueue, it injects a completion IRQ into the guest. Finally, a vCPU in the guest handles the interrupt and processes the completion. These different stages can potentially be processed by different physical cores in the host, perhaps after having transitioned from idle to running states.
On the other hand, filesystem operations tend to be synchronous by nature, which implies some workloads have nothing to do until the request has been processed. Combined with the fact that modern cores have deep pipelines and tend to aggressively go to sleep when they're idle, they're many workloads that can benefit from virtio-fs operating in a synchronous way.
Here, we implement the experimental VIRTIO_FS_F_SYNC feature. When negotiated, the device creates a dedicated queue per configured vCPU. After a vCPU in the guest triggers a vm-exit to kick the virtqueue, it's diverted to the device implementation to process the request by itself. Once finished, it returns to the guest and immediately checks the virtqueue, finding the reply to its own request.
While still experimental, real world benchmarks show very interesting performance improvements coming from this feature. For instance, building libkrunfw inside a VM with 8 vCPUs and 8192 GB on an M3 Pro is ~38% faster when using synchronous queues:
- libkrun with normal queues
root@localhost:/root/libkrunfw-5.6.2# time make -j8
real 7m43.826s
user 12m47.002s
sys 25m2.745s- libkrun with per-cpu synchronous queues
root@localhost:/root/libkrunfw-5.6.2# time make -j8
real 5m34.348s
user 13m16.870s
sys 11m3.636sPure synchronous I/O benchmarks also show a significant improvement:
- libkrun with normal queues, 1 job, best of 3 runs
# fio --name=virtiofs-test --directory=/root --rw=randrw --bs=4k --size=1G --numjobs=1 --time_based --runtime=60 --group_reporting --ioengine=pvsync --iodepth=1 --direct=1
virtiofs-test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync, iodepth=1
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=42.4MiB/s,w=43.0MiB/s][r=10.8k,w=11.0k IOPS][eta 00m:00s]
virtiofs-test: (groupid=0, jobs=1): err= 0: pid=118: Fri Oct 16 11:08:02 2026
read: IOPS=10.7k, BW=41.7MiB/s (43.8MB/s)(2505MiB/60001msec)
(...)
write: IOPS=10.7k, BW=41.7MiB/s (43.8MB/s)(2504MiB/60001msec); 0 zone resets- libkrun with normal queues, 4 jobs, best of 3 runs
# fio --name=virtiofs-test --directory=/root --rw=randrw --bs=4k --size=1G --numjobs=4 --time_based --runtime=60 --group_reporting --ioengine=pvsync --iodepth=1 --direct=1
virtiofs-test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync, iodepth=1
...
fio-3.39
Starting 4 processes
Jobs: 4 (f=4): [f(4)][100.0%][r=106MiB/s,w=106MiB/s][r=27.2k,w=27.1k IOPS][eta 00m:00s]
virtiofs-test: (groupid=0, jobs=4): err= 0: pid=121: Fri Oct 16 11:11:01 2026
read: IOPS=30.6k, BW=120MiB/s (125MB/s)(7180MiB/60001msec)
(...)
write: IOPS=30.6k, BW=120MiB/s (125MB/s)(7180MiB/60001msec); 0 zone resets- libkrun with per-cpu synchronous queues, 1 jobs, best of 3 runs
# fio --name=virtiofs-test --directory=/root --rw=randrw --bs=4k --size=1G --numjobs=1 --time_based --runtime=60 --group_reporting --ioengine=pvsync --iodepth=1 --direct=1
virtiofs-test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync, iodepth=1
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=54.1MiB/s,w=54.5MiB/s][r=13.9k,w=13.9k IOPS][eta 00m:00s]
virtiofs-test: (groupid=0, jobs=1): err= 0: pid=115: Fri Oct 16 10:51:35 2026
read: IOPS=14.1k, BW=55.2MiB/s (57.9MB/s)(3312MiB/60001msec)
(...)
write: IOPS=14.1k, BW=55.1MiB/s (57.8MB/s)(3307MiB/60001msec); 0 zone resets- libkrun with per-cpu synchronous queues, 4 jobs, best of 3 runs
# fio --name=virtiofs-test --directory=/root --rw=randrw --bs=4k --size=1G --numjobs=4 --time_based --runtime=60 --group_reporting --ioengine=pvsync --iodepth=1 --direct=1
virtiofs-test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync, iodepth=1
...
fio-3.39
Starting 4 processes
Jobs: 4 (f=4): [f(4)][100.0%][r=150MiB/s,w=149MiB/s][r=38.3k,w=38.2k IOPS][eta 00m:00s]
virtiofs-test: (groupid=0, jobs=4): err= 0: pid=119: Fri Oct 16 10:55:25 2026
read: IOPS=38.2k, BW=149MiB/s (156MB/s)(8947MiB/60002msec)
(...)
write: IOPS=38.2k, BW=149MiB/s (156MB/s)(8947MiB/60002msec); 0 zone resetsA synthetic benchmark evaluating the cost of the unlinkat syscall shows an improvement alignment with the previous tests:
- libkrun with normal queues
1. Creating 50000 files in 'unlinkat_bench_phased_tmp'...
2. Unlinking all 50000 files and measuring latency...
--- Statistics: Batch Unlink Phase ---
Files Unlinked: 50000
Average: 161457.91 ns (161.458 µs)
Median: 152625.00 ns (152.625 µs)
Std Dev: 32368.93 ns (32.369 µs)
Min: 113875 ns (113.875 µs)
Max: 2902583 ns (2902.583 µs)- libkrun with per-cpu synchronous queues
root@localhost:/# root/foobar
1. Creating 50000 files in 'unlinkat_bench_phased_tmp'...
2. Unlinking all 50000 files and measuring latency...
--- Statistics: Batch Unlink Phase ---
Files Unlinked: 50000
Average: 103191.24 ns (103.191 µs)
Median: 95708.00 ns (95.708 µs)
Std Dev: 32229.44 ns (32.229 µs)
Min: 72166 ns (72.166 µs)
Max: 3456917 ns (3456.917 µs)Other virtio device operations, such as read requests on virtio-blk, could also potentially benefit from using synchronous queues.
Activity
3This pull request is a draft
Merging is unavailable until it is marked ready for review.
Linux support for this feature can be found in this branch https://github.com/slp/linux/tree/virtio-fs-sync