Optimization: Detect bfq idling on the disks a sandbox uses, and turn it off in --host-settings=adjust

Open

The idea

On a disk with the bfq I/O scheduler, the default slice_idle (8 ms) makes bfq keep the disk idle after a synchronous request completes, waiting for the same process or cgroup to send the next one. gVisor turns application I/O, including O_DIRECT and AIO (at least before #15873), into synchronous host reads and writes from a few Sentry and gofer threads, which is the pattern penalized the most. Some GCP node images enable bfq with default idling (GKE COS node pools), and sandboxes there can read several times slower than runc does on the same disk.

runsc could add a host setting next to the existing THP and max_map_count checks. For each disk that holds the sandbox root filesystem (resolving an overlay rootfs to its upper or lower directory through mountinfo), the source of a gofer mount, or the image being restored, it would check whether bfq is active with a non-zero slice_idle. Like the other host settings, the default check mode would only log a warning, and only adjust/enforce would write slice_idle=0.

Performance impact

With --host-settings=adjust on a bfq disk, single-container fio_test reads get faster: up to 3x on the disk-backed bind mount (random 4K +151%, 64K-1M sequential +124-195%, 4K O_DIRECT sequential +12%) and up to 5.8x on the rootfs. Checkpoint/restore is 3-9% faster (startup_test), and a cold-cache Ruby run is 6-11% faster. Writes keep the same bandwidth, and Bazel builds don't change.
With 16 concurrent containers, the disk reaches its provisioned IOPS limit: runsc total +16%, and the slowest container gets 2.3x more IOPS. runc gets +18% there too.
Neighbours and weights: when a runc container and a runsc container read at the same time, bfq idling gives runc about 95% of the disk. With slice_idle=0 they split it about evenly, so runc loses 49%, which is expected and fair.
Default flags (warning only): no change in startup_test or size_test (n=16). The check itself takes about 0.2 ms per sandbox start, mostly from scanning /proc/self/mountinfo to resolve overlay disks, which grows to about 3.5 ms on hosts with ~4000 mounts. Only resolving disks if some /sys/block/*/queue/scheduler shows [bfq] keeps it at a flat ~0.1 ms on hosts without bfq, regardless of mount-table size.

Beat the autoresearch

The following benchmark results were benchmarked using this patch: https://github.com/google/gvisor/compare/master...EtiennePerot:gvisor:autoresearch/node-io-scheduler

Add value by beating these benchmark numbers, or reducing any regressions they cause, or both

goos: linux
goarch: amd64
cpu: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
pkg: 16 concurrent fio containers on one disk
                          │    before    │                after                 │
                          │     iops     │     iops      vs base                │
Multi/c16/runsc-total-16    8.349k ± ∞ ¹   9.695k ± ∞ ¹   +16.13% (p=0.008 n=5)
Multi/c16/runsc-median-16    495.1 ± ∞ ¹    606.3 ± ∞ ¹   +22.46% (p=0.008 n=5)
Multi/c16/runsc-min-16       255.7 ± ∞ ¹    585.6 ± ∞ ¹  +129.02% (p=0.008 n=5)
Multi/c16/runc-total-16     8.208k ± ∞ ¹   9.692k ± ∞ ¹   +18.08% (p=0.008 n=5)
Multi/c16/runc-median-16     525.9 ± ∞ ¹    595.3 ± ∞ ¹   +13.20% (p=0.008 n=5)
Multi/c16/runc-min-16        465.6 ± ∞ ¹    506.4 ± ∞ ¹    +8.76% (p=0.008 n=5)
geomean                     1.134k         1.469k         +29.58%
¹ need >= 6 samples for confidence interval at level 0.95

pkg: gvisor.dev/gvisor/test/benchmarks/base
                   │   before    │               after                │
                   │   sec/op    │   sec/op     vs base               │
CheckpointEmpty-16   69.65m ± 4%   63.71m ± 2%  -8.53% (p=0.000 n=16)
RestoreEmpty-16      204.0m ± 3%   190.5m ± 2%  -6.63% (p=0.000 n=16)
CheckpointNginx-16   114.0m ± 3%   111.4m ± 2%       ~ (p=0.061 n=16)
RestoreNginx-16      257.2m ± 3%   249.8m ± 1%  -2.90% (p=0.001 n=16)
geomean              142.9m        135.6m       -5.12%

pkg: gvisor.dev/gvisor/test/benchmarks/base (default flags)
                   │   before    │               after                │
                   │   sec/op    │   sec/op     vs base               │
CheckpointEmpty-16   59.03m ± 1%   60.92m ± 4%       ~ (p=0.110 n=16)
RestoreEmpty-16      194.6m ± 4%   194.5m ± 4%       ~ (p=0.780 n=16)
CheckpointNginx-16   96.29m ± 5%   94.76m ± 3%       ~ (p=0.445 n=16)
RestoreNginx-16      237.4m ± 3%   240.2m ± 2%       ~ (p=0.515 n=16)
StartupEmpty-16      193.7m ± 2%   190.8m ± 3%       ~ (p=0.305 n=16)
StartupNginx-16      539.0m ± 5%   542.5m ± 4%       ~ (p=0.809 n=16)
StartupNode-16       612.1m ± 2%   609.0m ± 3%       ~ (p=0.287 n=16)
SizeEmpty-16         449.7m ± 4%   443.8m ± 3%       ~ (p=0.254 n=16)
SizeNginx-16         876.3m ± 3%   873.3m ± 2%       ~ (p=0.590 n=16)
SizeNode-16          955.0m ± 2%   956.9m ± 3%       ~ (p=0.724 n=16)
geomean              302.0m        302.0m       -0.02%

             │            before            │                        after                        │
             │ average_container_size.bytes │ average_container_size.bytes  vs base               │
SizeEmpty-16                   20.80M ± 17%                   19.89M ± 25%       ~ (p=0.171 n=16)
SizeNginx-16                   32.66M ± 10%                   33.99M ± 10%       ~ (p=0.287 n=16)
SizeNode-16                    58.14M ±  6%                   60.77M ±  5%       ~ (p=0.287 n=16)
geomean                        34.06M                         34.51M        +1.32%

pkg: gvisor.dev/gvisor/test/benchmarks/fs
                                                                            │    before     │                after                │
                                                                            │    sec/op     │   sec/op     vs base                │
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16     48.48m ± 40%   21.44m ± 1%  -55.78% (p=0.000 n=16)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16      49.00m ± 47%   31.45m ± 0%  -35.81% (p=0.000 n=16)
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16      6.967m ±  3%   6.768m ± 1%   -2.86% (p=0.002 n=16)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16       13.43m ±  2%   13.17m ± 2%   -1.96% (p=0.017 n=16)
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16    10.207m ± 12%   8.254m ± 1%  -19.14% (p=0.000 n=16)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16      17.52m ± 13%   14.04m ± 1%  -19.86% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16        11.446m ±  7%   8.156m ± 2%  -28.74% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.64K/directIO.false/filesystem.bindfs-16       10.495m ± 14%   5.971m ± 2%  -43.11% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.bindfs-16      6.787m ± 10%   4.879m ± 2%  -28.12% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16          15.18m ±  5%   13.29m ± 1%  -12.43% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.bindfs-16         9.513m ± 16%   6.416m ± 2%  -32.56% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.bindfs-16       7.070m ±  8%   5.134m ± 1%  -27.38% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16          6.956m ±  5%   6.753m ± 1%        ~ (p=0.080 n=16)
FioRead/libaio/jobs.1/blockSize.64K/directIO.false/filesystem.tmpfs-16         4.296m ±  6%   4.221m ± 1%        ~ (p=0.094 n=16)
FioRead/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.tmpfs-16       4.115m ±  4%   4.055m ± 2%        ~ (p=0.323 n=16)
FioRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16           13.01m ±  3%   12.93m ± 1%        ~ (p=0.752 n=16)
FioRead/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.tmpfs-16          5.093m ±  5%   5.158m ± 1%        ~ (p=0.696 n=16)
FioRead/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.tmpfs-16        4.232m ±  4%   4.322m ± 1%        ~ (p=0.160 n=16)
FioRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16        10.771m ± 11%   8.091m ± 1%  -24.88% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.64K/directIO.false/filesystem.rootfs-16        9.169m ± 15%   6.354m ± 3%  -30.71% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.rootfs-16      5.654m ±  9%   5.118m ± 3%   -9.48% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16          14.37m ±  3%   13.38m ± 2%   -6.90% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.rootfs-16        10.344m ± 14%   6.625m ± 2%  -35.96% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.rootfs-16      11.321m ± 13%   5.433m ± 4%  -52.01% (p=0.000 n=16)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16    7.680m ±  5%   7.131m ± 1%   -7.15% (p=0.000 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16     14.74m ±  2%   14.18m ± 1%   -3.82% (p=0.000 n=16)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16     7.976m ±  5%   7.293m ± 1%   -8.56% (p=0.000 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16      14.42m ±  2%   14.07m ± 1%   -2.49% (p=0.002 n=16)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16    7.735m ±  1%   7.303m ± 1%   -5.57% (p=0.000 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16     14.23m ±  2%   13.60m ± 1%   -4.48% (p=0.000 n=16)
FioWrite/sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16          5.238m ±  5%   4.795m ± 1%   -8.46% (p=0.000 n=16)
FioWrite/sync/jobs.1/blockSize.64K/directIO.false/filesystem.bindfs-16         4.910m ±  7%   4.276m ± 2%  -12.91% (p=0.000 n=16)
FioWrite/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.bindfs-16     4.633m ±  5%   4.184m ± 1%   -9.70% (p=0.000 n=16)
FioWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16         14.36m ±  3%   13.92m ± 1%   -3.11% (p=0.000 n=16)
FioWrite/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.bindfs-16        7.434m ±  4%   6.837m ± 1%   -8.03% (p=0.000 n=16)
FioWrite/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.bindfs-16      6.207m ±  7%   5.771m ± 4%   -7.03% (p=0.003 n=16)
FioWrite/sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16           5.213m ±  8%   4.817m ± 1%   -7.61% (p=0.007 n=16)
FioWrite/sync/jobs.1/blockSize.64K/directIO.false/filesystem.tmpfs-16          4.806m ±  7%   4.325m ± 2%  -10.01% (p=0.000 n=16)
FioWrite/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.tmpfs-16      4.588m ±  6%   4.339m ± 1%   -5.42% (p=0.000 n=16)
FioWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16          14.25m ±  2%   13.88m ± 1%   -2.60% (p=0.004 n=16)
FioWrite/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.tmpfs-16         6.330m ±  6%   5.790m ± 1%   -8.52% (p=0.000 n=16)
FioWrite/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.tmpfs-16       5.481m ±  7%   5.118m ± 1%   -6.63% (p=0.001 n=16)
FioWrite/sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16          5.463m ±  7%   4.986m ± 1%   -8.75% (p=0.000 n=16)
FioWrite/sync/jobs.1/blockSize.64K/directIO.false/filesystem.rootfs-16         4.836m ±  7%   4.227m ± 1%  -12.59% (p=0.000 n=16)
FioWrite/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.rootfs-16     4.440m ±  9%   4.234m ± 1%   -4.63% (p=0.000 n=16)
FioWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16         13.71m ±  2%   13.54m ± 2%        ~ (p=0.138 n=16)
FioWrite/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.rootfs-16        7.396m ±  3%   6.829m ± 0%   -7.67% (p=0.000 n=16)
FioWrite/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.rootfs-16      6.431m ±  6%   5.776m ± 3%  -10.20% (p=0.000 n=16)
RubyNoOpTest/page_cache.clean/filesystem.bindfs-16                             730.1m ±  5%   651.0m ± 2%  -10.84% (p=0.000 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.bindfs-16                             246.2m ±  2%   246.7m ± 2%        ~ (p=0.642 n=16)
RubyNoOpTest/page_cache.clean/filesystem.tmpfs-16                              652.2m ±  6%   613.9m ± 3%   -5.87% (p=0.000 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.tmpfs-16                              246.1m ±  2%   244.5m ± 4%        ~ (p=0.539 n=16)
RubyNoOpTest/page_cache.clean/filesystem.rootfs-16                             672.9m ±  7%   617.8m ± 2%   -8.19% (p=0.000 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.rootfs-16                             248.4m ±  2%   249.7m ± 3%        ~ (p=0.564 n=16)
BuildABSL/page_cache.clean/filesystem.bindfs-16                                 31.63 ±  3%    31.34 ± 3%        ~ (p=0.287 n=16)
BuildABSL/page_cache.dirty/filesystem.bindfs-16                                 30.75 ±  1%    31.22 ± 4%        ~ (p=0.402 n=16)
BuildABSL/page_cache.clean/filesystem.tmpfs-16                                  31.41 ±  2%    31.49 ± 2%        ~ (p=0.926 n=16)
BuildABSL/page_cache.dirty/filesystem.tmpfs-16                                  30.84 ±  1%    30.69 ± 2%        ~ (p=0.564 n=16)
BuildABSL/page_cache.clean/filesystem.rootfs-16                                 30.97 ±  4%    31.58 ± 1%   +1.95% (p=0.021 n=16)
BuildABSL/page_cache.dirty/filesystem.rootfs-16                                 30.77 ±  1%    31.06 ± 3%        ~ (p=0.669 n=16)
geomean                                                                        28.57m         25.08m       -12.24%

                                                                            │               before                │                            after                             │
                                                                            │ bandwidth_all_jobs.bytes_per_second │ bandwidth_all_jobs.bytes_per_second  vs base                 │
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16                           23.98M ± 75%                          60.24M ±  1%  +151.23% (p=0.000 n=16)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16                            191.4M ± 35%                          308.4M ±  0%   +61.11% (p=0.000 n=16)
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16                            358.9M ±  3%                          381.8M ±  3%    +6.39% (p=0.000 n=16)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16                             933.2M ±  3%                          931.4M ±  1%         ~ (p=0.642 n=16)
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16                           186.4M ± 11%                          246.5M ±  2%   +32.22% (p=0.000 n=16)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16                            630.4M ± 11%                          850.8M ±  1%   +34.96% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16                               151.1M ± 15%                          255.2M ±  1%   +68.87% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.64K/directIO.false/filesystem.bindfs-16                              184.3M ± 25%                          544.6M ±  3%  +195.45% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.bindfs-16                            496.2M ± 38%                         1198.5M ±  7%  +141.53% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16                                827.6M ±  6%                          924.5M ±  2%   +11.71% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.bindfs-16                               1.647G ± 30%                          3.692G ±  8%  +124.14% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.bindfs-16                             3.135G ± 19%                          8.981G ±  8%  +186.45% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16                                375.4M ±  4%                          378.1M ±  3%         ~ (p=0.609 n=16)
FioRead/libaio/jobs.1/blockSize.64K/directIO.false/filesystem.tmpfs-16                               3.290G ± 24%                          4.067G ±  6%         ~ (p=0.118 n=16)
FioRead/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.tmpfs-16                             12.81G ± 64%                          15.85G ± 15%         ~ (p=0.213 n=16)
FioRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16                                 953.9M ±  1%                          955.5M ±  1%         ~ (p=0.788 n=16)
FioRead/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.tmpfs-16                                8.723G ±  5%                          8.713G ±  5%         ~ (p=0.809 n=16)
FioRead/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.tmpfs-16                              60.58G ±  6%                          60.72G ±  4%         ~ (p=0.897 n=16)
FioRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16                               168.3M ± 22%                          255.2M ±  3%   +51.61% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.64K/directIO.false/filesystem.rootfs-16                              219.1M ± 10%                          449.6M ± 12%  +105.19% (p=0.000 n=16)
FioRead/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.rootfs-16                            628.7M ± 25%                          965.6M ±  5%   +53.60% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16                                842.9M ±  5%                          919.6M ±  2%    +9.11% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.rootfs-16                               1.413G ± 23%                          3.492G ±  5%  +147.17% (p=0.000 n=16)
FioRead/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.rootfs-16                             1.206G ± 24%                          6.959G ± 14%  +477.03% (p=0.000 n=16)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16                          326.2M ±  1%                          326.6M ±  1%         ~ (p=0.978 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16                           843.9M ±  1%                          841.0M ±  1%         ~ (p=0.491 n=16)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16                           347.3M ±  1%                          346.8M ±  1%         ~ (p=0.933 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16                            876.2M ±  3%                          875.7M ±  2%         ~ (p=0.802 n=16)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16                          318.8M ±  3%                          315.8M ±  1%         ~ (p=0.365 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16                           883.2M ±  1%                          888.0M ±  1%         ~ (p=0.515 n=16)
FioWrite/sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16                                1.133G ±  4%                          1.137G ±  4%         ~ (p=0.772 n=16)
FioWrite/sync/jobs.1/blockSize.64K/directIO.false/filesystem.bindfs-16                               2.632G ±  4%                          2.658G ±  3%         ~ (p=0.312 n=16)
FioWrite/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.bindfs-16                           3.121G ±  4%                          3.050G ±  2%         ~ (p=0.316 n=16)
FioWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16                               854.3M ±  3%                          857.0M ±  1%         ~ (p=0.445 n=16)
FioWrite/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.bindfs-16                              3.107G ±  1%                          3.092G ±  0%         ~ (p=0.056 n=16)
FioWrite/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.bindfs-16                            5.469G ± 11%                          4.983G ± 17%         ~ (p=0.270 n=16)
FioWrite/sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16                                 1.517G ±  5%                          1.517G ±  4%         ~ (p=0.673 n=16)
FioWrite/sync/jobs.1/blockSize.64K/directIO.false/filesystem.tmpfs-16                                5.836G ±  4%                          5.836G ±  4%         ~ (p=0.950 n=16)
FioWrite/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.tmpfs-16                            7.457G ±  5%                          7.064G ±  6%         ~ (p=0.286 n=16)
FioWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16                                881.5M ±  1%                          887.0M ±  2%         ~ (p=0.780 n=16)
FioWrite/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.tmpfs-16                               5.797G ±  9%                          5.801G ±  1%         ~ (p=0.669 n=16)
FioWrite/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.tmpfs-16                             11.42G ± 13%                          11.46G ±  2%         ~ (p=0.429 n=16)
FioWrite/sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16                                965.6M ±  3%                          945.2M ±  3%         ~ (p=0.426 n=16)
FioWrite/sync/jobs.1/blockSize.64K/directIO.false/filesystem.rootfs-16                               2.739G ±  6%                          2.684G ±  4%         ~ (p=0.345 n=16)
FioWrite/libaio/jobs.1/blockSize.1024K/directIO.false/filesystem.rootfs-16                           2.950G ±  3%                          2.983G ±  2%         ~ (p=0.554 n=16)
FioWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16                               896.1M ±  1%                          895.8M ±  3%         ~ (p=0.305 n=16)
FioWrite/libaio/jobs.8/blockSize.64K/directIO.true/filesystem.rootfs-16                              3.074G ±  1%                          3.047G ±  1%         ~ (p=0.086 n=16)
FioWrite/libaio/jobs.8/blockSize.1024K/directIO.true/filesystem.rootfs-16                            4.782G ±  3%                          4.838G ± 16%         ~ (p=0.539 n=16)
geomean                                                                                              1.186G                                1.493G         +25.90%

pkg: gvisor.dev/gvisor/test/benchmarks/fs, runc (slice_idle 8 -> 0)
                                                                            │    before    │                after                │
                                                                            │    sec/op    │    sec/op     vs base               │
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16    115.4m ± 10%   112.5m ±  4%       ~ (p=0.529 n=10)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16     217.3m ±  0%   217.0m ±  0%  -0.12% (p=0.011 n=10)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16   5.940m ±  7%   5.501m ±  5%  -7.38% (p=0.000 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16    452.6m ±  5%   514.8m ± 11%       ~ (p=0.110 n=16)
geomean                                                                       90.62m         91.19m        +0.63%

                                                                            │               before                │                           after                            │
                                                                            │ bandwidth_all_jobs.bytes_per_second │ bandwidth_all_jobs.bytes_per_second  vs base               │
FioRandRead/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16                           9.522M ± 12%                           9.750M ± 4%       ~ (p=0.631 n=10)
FioRandRead/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16                            42.78M ±  5%                           44.68M ± 7%       ~ (p=0.353 n=10)
FioRandWrite/libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16                          2.097G ±  0%                           2.097G ± 2%       ~ (p=0.731 n=16)
FioRandWrite/libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16                           22.54M ±  5%                           21.83M ± 9%       ~ (p=0.239 n=16)
geomean                                                                                              66.24M                                 66.83M       +0.88%

pkg: two concurrent fio containers on one disk (slice_idle 8 -> 0)
                            │    before    │                after                 │
                            │     iops     │     iops      vs base                │
Mixed/mixed-rr/runc-16        9.026k ± ∞ ¹   4.617k ± ∞ ¹   -48.85% (p=0.008 n=5)
Mixed/mixed-rr/runsc-16        501.7 ± ∞ ¹   5277.5 ± ∞ ¹  +951.92% (p=0.008 n=5)
Mixed/weights/runc-w100-16    1.376k ± ∞ ¹   4.825k ± ∞ ¹  +250.70% (p=0.008 n=5)
Mixed/weights/runc-w1000-16   7.818k ± ∞ ¹   4.865k ± ∞ ¹   -37.76% (p=0.008 n=5)
geomean                       2.642k         4.890k         +85.12%
¹ need >= 6 samples for confidence interval at level 0.95

Benchmark environment

GCE c4-standard-16, COS cos-129-19506-448-8 (kernel 6.12.105+), a single hyperdisk-balanced boot disk (200 GB, 9600 IOPS / 2400 MB/s). The disk was switched to bfq with slice_idle=8, low_latency=1 before every run, and THP and max_map_count were preset so that adjust only changes slice_idle. before is master at 2511bc60218a with default flags. In the test/benchmarks/fs and base blocks, after is the patch with --host-settings=adjust, which sets slice_idle=0 when the container starts. The base (default flags) block runs the patch with default flags, where it only warns. Blocks marked slice_idle 8 -> 0 use master runsc or runc in both arms and compare the two settings directly. The concurrent-container blocks run fio randread 4K O_DIRECT libaio qd4 from docker containers on separate files on the disk for 20 s (16 containers with one job each, or two containers with four jobs each). The upstream binaries add an all-jobs fio bandwidth metric and write real data before the read tests (fallocated extents are never read from disk). The fs fusefs variants were skipped because the FUSE server binary was not available. The runc fs rows are bindfs only. Runs alternate AB/BA, with caches dropped before each one. The measured branch build differs from the linked patch only in a comment, the warning text and tests.

Activity

No activity yet.