#16154
Optimization: Let gofers exit once their container no longer needs them
The idea
Each container keeps a gofer process alive for its whole lifetime, even though with directfs the Sentry opens host files itself and the gofer is only still used for a few fallbacks (bind/connect of host sockets, xattrs on symlinks and sockets, opens of file-backed mount roots). Every idle gofer costs 2.2-3.5 MiB of non-reclaimable memory per container, growing with the host's core count.
runsc could close a container's lisafs connections shortly after it starts or is restored (e.g. 1 s later), letting its gofer exit, whenever nothing it serves can still need the gofer: directfs on, --host-uds=none, no device gofer, and every gofer mount either read-only, overlaid, or a regular-file bind (like docker's /etc/hosts). File roots would be pre-opened before release; the remaining fallbacks would behave as under --host-uds=none (bind/connect return EPERM) and report "no xattrs" on symlinks and sockets, which keeps overlay copy-up working. Release would be all-or-nothing per container, since a gofer serving even one mount stays alive. Containers with a writable directory bind mount would keep their gofer. Since runsc would then act on gofer PIDs after the gofer exited, it should also record the gofer's start time and stop signaling/re-scoring a PID that has been reused. That also fixes a latent bug on master, where runsc delete SIGKILLs an unrelated process if the gofer died and its PID was reused.
Performance impact
- Per-container non-reclaimable memory with docker defaults: -2.35 MiB (-21%) on 4 vCPUs, -2.6 MiB (-20%) on 16, -3.2 MiB (-20%) on 48. Restored sandboxes: -2.2 to -3.5 MiB (-14% to -18%).
//test/benchmarks/base:usage_testmemory:SizeEmpty-17%,SizeNginx-11%,SizeNode-6%.- Container start,
docker run, checkpoint and restore latency unchanged; fio, rubydev, nginx, node and httpd unchanged.
Beat the autoresearch
The following benchmark results were benchmarked using this patch: https://github.com/google/gvisor/compare/master...EtiennePerot:gvisor:autoresearch/gofer-release
Add value by beating these benchmark numbers, or reducing any regressions they cause, or both
goos: linux
goarch: amd64
pkg: checkpoint/restore harness (runsc restore of one alpine container, overlay rootfs + file binds)
│ before │ after │
│ B │ B vs base │
RestoredNonReclaimableMem/vcpus=4 12.80Mi ± 2% 10.50Mi ± 3% -17.98% (p=0.000 n=10)
RestoredNonReclaimableMem/vcpus=16 15.40Mi ± 2% 13.23Mi ± 2% -14.07% (p=0.000 n=16)
RestoredNonReclaimableMem/vcpus=48 20.22Mi ± 2% 16.77Mi ± 2% -17.07% (p=0.000 n=26)
geomean 15.86Mi 13.26Mi -16.39%
│ before │ after │
│ sec/op │ sec/op vs base │
Restore/vcpus=4 39.20m ± 3% 39.30m ± 7% ~ (p=0.987 n=10)
Restore/vcpus=16 40.10m ± 3% 39.95m ± 4% ~ (p=0.570 n=16)
Restore/vcpus=48 51.85m ± 3% 52.90m ± 6% ~ (p=0.178 n=26)
geomean 43.36m 43.63m +0.63%
pkg: docker run memory harness (N idle alpine containers, per-container average)
│ before │ after │
│ B │ B vs base │
ContainerNonReclaimableMem/vcpus=4/containers=20 11.146Mi ± 1% 8.795Mi ± 1% -21.09% (p=0.002 n=6)
ContainerAnonMem/vcpus=4/containers=20 9.656Mi ± 1% 7.638Mi ± 1% -20.90% (p=0.002 n=6)
ContainerNonReclaimableMem/vcpus=16/containers=30 12.79Mi ± 1% 10.18Mi ± 1% -20.43% (p=0.002 n=6)
ContainerAnonMem/vcpus=16/containers=30 11.208Mi ± 1% 8.944Mi ± 1% -20.20% (p=0.002 n=6)
ContainerNonReclaimableMem/vcpus=48/containers=20 15.88Mi ± 1% 12.71Mi ± 1% -19.95% (p=0.002 n=6)
ContainerAnonMem/vcpus=48/containers=20 14.15Mi ± 1% 11.30Mi ± 1% -20.14% (p=0.002 n=6)
geomean 12.30Mi 9.786Mi -20.45%
│ before │ after │
│ sec/op │ sec/op vs base │
DockerRunAll/vcpus=4/containers=20 3.938 ± 3% 3.949 ± 2% ~ (p=0.818 n=6)
DockerRunAll/vcpus=16/containers=30 6.005 ± 5% 5.974 ± 1% ~ (p=0.413 n=6)
DockerRunAll/vcpus=48/containers=20 4.437 ± 4% 4.449 ± 2% ~ (p=0.937 n=6)
geomean 4.716 4.717 +0.01%
pkg: gvisor.dev/gvisor/test/benchmarks/base
│ before │ after │
│ sec/op │ sec/op vs base │
StartupEmpty-16 171.8m ± 1% 172.4m ± 2% ~ (p=0.956 n=16)
StartupNginx-16 461.9m ± 2% 462.0m ± 2% ~ (p=0.926 n=16)
StartupNode-16 560.4m ± 1% 557.0m ± 1% ~ (p=0.323 n=16)
CheckpointEmpty-16 56.54m ± 1% 55.89m ± 2% ~ (p=0.254 n=16)
RestoreEmpty-16 185.4m ± 2% 183.8m ± 2% ~ (p=0.224 n=16)
CheckpointNginx-16 91.32m ± 1% 91.35m ± 2% ~ (p=0.956 n=16)
RestoreNginx-16 232.8m ± 2% 231.8m ± 3% ~ (p=0.724 n=16)
geomean 192.8m 192.1m -0.38%
pkg: gvisor.dev/gvisor/test/benchmarks/base (size_test)
│ before │ after │
│ sec/op │ sec/op vs base │
SizeEmpty-16 349.3m ± 1% 348.7m ± 1% ~ (p=0.956 n=16)
SizeNginx-16 705.0m ± 1% 704.1m ± 1% ~ (p=0.926 n=16)
SizeNode-16 802.2m ± 1% 797.4m ± 1% ~ (p=0.752 n=16)
geomean 582.4m 580.7m -0.29%
│ before │ after │
│ average_container_size.bytes │ average_container_size.bytes vs base │
SizeEmpty-16 21.58M ± 17% 22.24M ± 22% ~ (p=0.491 n=16)
SizeNginx-16 35.00M ± 17% 28.79M ± 17% -17.73% (p=0.004 n=16)
SizeNode-16 57.23M ± 7% 56.54M ± 3% ~ (p=0.669 n=16)
geomean 35.09M 33.08M -5.73%
pkg: gvisor.dev/gvisor/test/benchmarks/base (usage_test)
│ before │ after │
│ sec/op │ sec/op vs base │
SizeEmpty-16 2.094 ± 0% 2.095 ± 0% ~ (p=0.491 n=16)
SizeNginx-16 2.135 ± 0% 2.132 ± 0% ~ (p=0.086 n=16)
SizeNode-16 2.109 ± 0% 2.109 ± 0% ~ (p=0.956 n=16)
geomean 2.113 2.112 -0.02%
│ before │ after │
│ average_container_memory_usage.bytes │ average_container_memory_usage.bytes vs base │
SizeEmpty-16 14.86M ± 1% 12.32M ± 1% -17.09% (p=0.000 n=16)
SizeNginx-16 24.05M ± 0% 21.51M ± 1% -10.56% (p=0.000 n=16)
SizeNode-16 48.85M ± 0% 46.10M ± 0% -5.63% (p=0.000 n=16)
geomean 25.94M 23.03M -11.22%
pkg: gvisor.dev/gvisor/test/benchmarks/fs
│ before │ after │
│ sec/op │ sec/op vs base │
RubyNoOpTest/page_cache.clean/filesystem.bindfs-16 599.2m ± 2% 597.9m ± 2% ~ (p=0.867 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.bindfs-16 253.9m ± 4% 247.2m ± 3% ~ (p=0.305 n=16)
RubyNoOpTest/page_cache.clean/filesystem.tmpfs-16 580.7m ± 2% 581.6m ± 3% ~ (p=1.000 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.tmpfs-16 246.3m ± 4% 249.8m ± 4% ~ (p=0.669 n=16)
RubyNoOpTest/page_cache.clean/filesystem.rootfs-16 578.4m ± 1% 579.5m ± 2% ~ (p=0.724 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.rootfs-16 247.7m ± 4% 251.1m ± 3% ~ (p=0.270 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 6.060m ± 3% 6.027m ± 2% ~ (p=0.838 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 4.670m ± 2% 4.683m ± 2% ~ (p=0.985 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 6.042m ± 3% 5.999m ± 2% ~ (p=0.402 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 2.839m ± 2% 2.839m ± 2% ~ (p=0.539 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 2.884m ± 1% 2.904m ± 1% ~ (p=0.110 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 3.050m ± 2% 3.024m ± 1% ~ (p=0.809 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 31.16m ± 1% 30.95m ± 1% ~ (p=0.254 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 12.86m ± 1% 12.89m ± 2% ~ (p=1.000 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 13.83m ± 1% 13.67m ± 1% ~ (p=0.051 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 14.02m ± 1% 14.03m ± 1% ~ (p=0.985 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 13.91m ± 2% 13.88m ± 1% ~ (p=0.642 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 13.42m ± 2% 13.56m ± 1% ~ (p=0.616 n=16)
geomean 28.86m 28.84m -0.10%
│ before │ after │
│ bandwidth.bytes_per_second │ bandwidth.bytes_per_second vs base │
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 253.0M ± 2% 255.0M ± 3% ~ (p=0.773 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 380.8M ± 2% 380.2M ± 2% ~ (p=0.831 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 254.0M ± 2% 257.5M ± 2% ~ (p=0.366 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 1.147G ± 2% 1.145G ± 3% ~ (p=0.830 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 1.525G ± 4% 1.521G ± 4% ~ (p=0.874 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 932.1M ± 3% 946.9M ± 2% ~ (p=0.199 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 38.52M ± 1% 38.81M ± 1% ~ (p=0.116 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 117.3M ± 1% 117.9M ± 2% ~ (p=0.845 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 106.1M ± 2% 108.1M ± 1% +1.93% (p=0.012 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 104.6M ± 1% 104.5M ± 1% ~ (p=0.933 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 108.8M ± 2% 109.6M ± 2% ~ (p=0.497 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 111.3M ± 3% 110.2M ± 1% ~ (p=0.509 n=16)
geomean 231.9M 233.1M +0.51%
│ before │ after │
│ bandwidth_all_jobs.bytes_per_second │ bandwidth_all_jobs.bytes_per_second vs base │
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 253.0M ± 2% 255.0M ± 3% ~ (p=0.773 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 380.8M ± 2% 380.2M ± 2% ~ (p=0.831 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 254.0M ± 2% 257.5M ± 2% ~ (p=0.366 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 1.147G ± 2% 1.145G ± 3% ~ (p=0.830 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 1.525G ± 4% 1.521G ± 4% ~ (p=0.874 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 932.1M ± 3% 946.9M ± 2% ~ (p=0.199 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 308.6M ± 1% 311.0M ± 1% ~ (p=0.108 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 936.7M ± 1% 940.2M ± 1% ~ (p=0.642 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 848.8M ± 2% 865.1M ± 1% +1.92% (p=0.023 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 836.6M ± 1% 838.3M ± 2% ~ (p=0.616 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 869.1M ± 2% 876.8M ± 2% ~ (p=0.445 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 890.3M ± 3% 880.2M ± 1% ~ (p=0.515 n=16)
geomean 655.9M 659.2M +0.51%
│ before │ after │
│ io_ops.ops_per_second │ io_ops.ops_per_second vs base │
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 61.77k ± 2% 62.27k ± 3% ~ (p=0.773 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 92.96k ± 2% 92.83k ± 2% ~ (p=0.831 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 62.00k ± 2% 62.86k ± 2% ~ (p=0.366 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 280.1k ± 2% 279.5k ± 3% ~ (p=0.830 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 372.4k ± 4% 371.3k ± 4% ~ (p=0.874 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 227.6k ± 3% 231.2k ± 2% ~ (p=0.199 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 9.404k ± 1% 9.476k ± 1% ~ (p=0.116 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 28.63k ± 1% 28.78k ± 2% ~ (p=0.845 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 25.89k ± 2% 26.39k ± 1% +1.93% (p=0.012 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 25.53k ± 1% 25.51k ± 1% ~ (p=0.933 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 26.55k ± 2% 26.76k ± 2% ~ (p=0.497 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 27.17k ± 3% 26.90k ± 1% ~ (p=0.509 n=16)
geomean 56.62k 56.90k +0.51%
│ before │ after │
│ io_ops_all_jobs.ops_per_second │ io_ops_all_jobs.ops_per_second vs base │
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 61.77k ± 2% 62.27k ± 3% ~ (p=0.773 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 92.96k ± 2% 92.83k ± 2% ~ (p=0.831 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 62.00k ± 2% 62.86k ± 2% ~ (p=0.366 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16 280.1k ± 2% 279.5k ± 3% ~ (p=0.830 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16 372.4k ± 4% 371.3k ± 4% ~ (p=0.874 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16 227.6k ± 3% 231.2k ± 2% ~ (p=0.199 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 75.34k ± 1% 75.92k ± 1% ~ (p=0.108 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 228.7k ± 1% 229.6k ± 1% ~ (p=0.649 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 207.2k ± 2% 211.2k ± 1% +1.92% (p=0.023 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16 204.2k ± 1% 204.7k ± 2% ~ (p=0.616 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16 212.2k ± 2% 214.1k ± 2% ~ (p=0.445 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16 217.4k ± 3% 214.9k ± 1% ~ (p=0.515 n=16)
geomean 160.1k 160.9k +0.51%
pkg: gvisor.dev/gvisor/test/benchmarks/network
│ before │ after │
│ sec/op │ sec/op vs base │
NginxDocSize/filesize.1Kb/concurrency.1/filesystem.tmpfs-16 416.0µ ± 3% 421.8µ ± 2% ~ (p=0.094 n=16)
NginxDocSize/filesize.1Kb/concurrency.100/filesystem.tmpfs-16 325.1µ ± 2% 326.1µ ± 2% ~ (p=0.956 n=16)
NginxDocSize/filesize.1Kb/concurrency.1000/filesystem.tmpfs-16 407.0µ ± 39% 391.5µ ± 3% ~ (p=0.061 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.tmpfs-16 524.7µ ± 2% 529.3µ ± 2% ~ (p=0.539 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.tmpfs-16 395.4µ ± 2% 403.0µ ± 4% ~ (p=0.270 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.tmpfs-16 571.4µ ± 21% 468.6µ ± 24% ~ (p=0.224 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.tmpfs-16 7.589m ± 1% 7.600m ± 1% ~ (p=0.564 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.tmpfs-16 7.007m ± 2% 7.074m ± 2% ~ (p=1.000 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.tmpfs-16 6.192m ± 3% 6.405m ± 5% +3.44% (p=0.032 n=16)
NginxDocSize/filesize.1Kb/concurrency.1/filesystem.bind-16 423.1µ ± 3% 424.0µ ± 2% ~ (p=0.445 n=16)
NginxDocSize/filesize.1Kb/concurrency.100/filesystem.bind-16 333.9µ ± 3% 338.6µ ± 4% ~ (p=0.119 n=16)
NginxDocSize/filesize.1Kb/concurrency.1000/filesystem.bind-16 410.6µ ± 4% 399.7µ ± 4% ~ (p=0.061 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.bind-16 546.7µ ± 2% 550.8µ ± 2% ~ (p=0.445 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.bind-16 411.1µ ± 3% 423.3µ ± 5% ~ (p=0.171 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.bind-16 581.7µ ± 21% 559.7µ ± 20% ~ (p=0.138 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.bind-16 8.399m ± 1% 8.383m ± 1% ~ (p=0.838 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.bind-16 7.978m ± 3% 7.956m ± 3% ~ (p=0.616 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.bind-16 7.458m ± 3% 7.478m ± 3% ~ (p=0.780 n=16)
Node/concurrency.1-16 1.682m ± 4% 1.685m ± 1% ~ (p=0.724 n=16)
Node/concurrency.5-16 1.372m ± 1% 1.368m ± 1% ~ (p=0.642 n=16)
Node/concurrency.10-16 1.343m ± 4% 1.352m ± 4% ~ (p=0.515 n=16)
Node/concurrency.25-16 1.281m ± 0% 1.282m ± 1% ~ (p=0.780 n=16)
Httpd/filesize.1Kb/concurrency.1-16 626.7µ ± 2% 627.2µ ± 1% ~ (p=0.985 n=16)
Httpd/filesize.1Kb/concurrency.100-16 5.402m ± 92% 5.379m ± 92% ~ (p=0.361 n=16)
Httpd/filesize.100Kb/concurrency.1-16 938.9µ ± 1% 947.7µ ± 1% ~ (p=0.160 n=16)
Httpd/filesize.100Kb/concurrency.100-16 5.421m ± 91% 5.425m ± 91% ~ (p=0.752 n=16)
Httpd/filesize.10Mb/concurrency.1-16 33.23m ± 2% 33.04m ± 2% ~ (p=0.491 n=16)
Httpd/filesize.10Mb/concurrency.100-16 8.394m ± 8% 7.729m ± 36% ~ (p=0.051 n=16)
geomean 1.532m 1.520m -0.84%
│ before │ after │
│ average_latency.s │ average_latency.s vs base │
NginxDocSize/filesize.1Kb/concurrency.1/filesystem.tmpfs-16 100.0µ ± 0% 100.0µ ± 0% ~ (p=1.000 n=16) ¹
NginxDocSize/filesize.1Kb/concurrency.100/filesystem.tmpfs-16 4.300m ± 16% 4.500m ± 24% ~ (p=0.948 n=16)
NginxDocSize/filesize.1Kb/concurrency.1000/filesystem.tmpfs-16 81.55m ± 25% 85.85m ± 24% ~ (p=0.583 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.tmpfs-16 200.0µ ± 0% 200.0µ ± 50% ~ (p=0.704 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.tmpfs-16 9.150m ± 5% 9.450m ± 7% ~ (p=0.355 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.tmpfs-16 102.3m ± 6% 104.1m ± 19% ~ (p=0.846 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.tmpfs-16 7.300m ± 1% 7.300m ± 1% ~ (p=0.611 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.tmpfs-16 452.0m ± 13% 465.8m ± 4% +3.06% (p=0.004 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.tmpfs-16 3.027 ± 4% 3.149 ± 6% +4.01% (p=0.022 n=16)
NginxDocSize/filesize.1Kb/concurrency.1/filesystem.bind-16 100.0µ ± 0% 100.0µ ± 0% ~ (p=1.000 n=16)
NginxDocSize/filesize.1Kb/concurrency.100/filesystem.bind-16 4.650m ± 16% 4.700m ± 17% ~ (p=0.948 n=16)
NginxDocSize/filesize.1Kb/concurrency.1000/filesystem.bind-16 86.95m ± 32% 66.00m ± 15% ~ (p=0.128 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.bind-16 300.0µ ± 0% 300.0µ ± 0% ~ (p=1.000 n=16) ¹
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.bind-16 10.15m ± 4% 10.15m ± 3% ~ (p=0.772 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.bind-16 107.6m ± 6% 106.7m ± 17% ~ (p=0.889 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.bind-16 8.100m ± 1% 8.100m ± 1% ~ (p=0.640 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.bind-16 524.9m ± 2% 531.5m ± 3% ~ (p=0.396 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.bind-16 3.631 ± 3% 3.683 ± 3% ~ (p=0.724 n=16)
Node/concurrency.1-16 1.000m ± 0% 1.100m ± 9% ~ (p=0.066 n=16)
Node/concurrency.5-16 3.750m ± 1% 3.700m ± 3% ~ (p=0.888 n=16)
Node/concurrency.10-16 7.300m ± 1% 7.200m ± 1% ~ (p=0.519 n=16)
Node/concurrency.25-16 16.20m ± 2% 16.25m ± 2% ~ (p=0.844 n=16)
Httpd/filesize.1Kb/concurrency.1-16 400.0µ ± 25% 400.0µ ± 25% ~ (p=1.000 n=16)
Httpd/filesize.1Kb/concurrency.100-16 50.40m ± 73% 40.10m ± 68% ~ (p=0.417 n=16)
Httpd/filesize.100Kb/concurrency.1-16 700.0µ ± 0% 700.0µ ± 0% ~ (p=1.000 n=16)
Httpd/filesize.100Kb/concurrency.100-16 60.15m ± 72% 49.95m ± 68% ~ (p=0.533 n=16)
Httpd/filesize.10Mb/concurrency.1-16 32.95m ± 1% 32.75m ± 2% ~ (p=0.508 n=16)
Httpd/filesize.10Mb/concurrency.100-16 367.3m ± 5% 350.8m ± 7% -4.48% (p=0.044 n=16)
geomean 13.65m 13.45m -1.51%
¹ all samples are equal
│ before │ after │
│ requests_per_second.QPS │ requests_per_second.QPS vs base │
NginxDocSize/filesize.1Kb/concurrency.1/filesystem.tmpfs-16 7.098k ± 1% 7.066k ± 1% ~ (p=0.596 n=16)
NginxDocSize/filesize.1Kb/concurrency.100/filesystem.tmpfs-16 19.72k ± 12% 18.69k ± 11% ~ (p=0.564 n=16)
NginxDocSize/filesize.1Kb/concurrency.1000/filesystem.tmpfs-16 7.406k ± 54% 8.383k ± 20% +13.20% (p=0.014 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.tmpfs-16 3.998k ± 1% 3.994k ± 0% ~ (p=0.570 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.tmpfs-16 8.206k ± 3% 8.306k ± 5% ~ (p=0.889 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.tmpfs-16 3.390k ± 3% 5.288k ± 38% ~ (p=0.305 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.tmpfs-16 136.6 ± 1% 136.6 ± 1% ~ (p=0.622 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.tmpfs-16 148.4 ± 3% 147.4 ± 1% ~ (p=0.993 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.tmpfs-16 169.2 ± 3% 163.5 ± 2% -3.34% (p=0.027 n=16)
NginxDocSize/filesize.1Kb/concurrency.1/filesystem.bind-16 6.747k ± 2% 6.689k ± 1% ~ (p=0.136 n=16)
NginxDocSize/filesize.1Kb/concurrency.100/filesystem.bind-16 17.15k ± 11% 16.86k ± 9% ~ (p=0.515 n=16)
NginxDocSize/filesize.1Kb/concurrency.1000/filesystem.bind-16 7.577k ± 54% 8.315k ± 8% ~ (p=0.056 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.bind-16 3.728k ± 0% 3.704k ± 1% -0.64% (p=0.002 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.bind-16 7.210k ± 3% 7.136k ± 1% ~ (p=0.381 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.bind-16 3.323k ± 68% 4.090k ± 38% ~ (p=0.239 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.bind-16 123.5 ± 1% 123.5 ± 1% ~ (p=0.831 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.bind-16 130.1 ± 2% 130.4 ± 2% ~ (p=0.662 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.bind-16 139.5 ± 2% 139.5 ± 3% ~ (p=0.802 n=16)
Node/concurrency.1-16 960.0 ± 1% 949.9 ± 1% ~ (p=0.057 n=16)
Node/concurrency.5-16 1.330k ± 1% 1.330k ± 1% ~ (p=0.919 n=16)
Node/concurrency.10-16 1.371k ± 1% 1.376k ± 1% ~ (p=0.717 n=16)
Node/concurrency.25-16 1.528k ± 1% 1.528k ± 1% ~ (p=0.788 n=16)
Httpd/filesize.1Kb/concurrency.1-16 2.824k ± 1% 2.830k ± 1% ~ (p=0.889 n=16)
Httpd/filesize.1Kb/concurrency.100-16 195.2 ± 2726% 195.9 ± 2804% ~ (p=0.116 n=16)
Httpd/filesize.100Kb/concurrency.1-16 1.502k ± 1% 1.488k ± 1% ~ (p=0.121 n=16)
Httpd/filesize.100Kb/concurrency.100-16 194.6 ± 1% 194.7 ± 2228% ~ (p=0.963 n=16)
Httpd/filesize.10Mb/concurrency.1-16 30.34 ± 1% 30.53 ± 2% ~ (p=0.491 n=16)
Httpd/filesize.10Mb/concurrency.100-16 123.2 ± 2% 134.3 ± 58% +9.05% (p=0.048 n=16)
geomean 1.183k 1.219k +2.99%
pkg: isolated re-run of nginx
│ before │ after │
│ sec/op │ sec/op vs base │
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.tmpfs-16 534.2µ ± 2% 532.2µ ± 1% ~ (p=0.926 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.tmpfs-16 399.9µ ± 1% 395.6µ ± 3% ~ (p=0.564 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.tmpfs-16 558.9µ ± 20% 448.5µ ± 8% ~ (p=0.102 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.tmpfs-16 7.644m ± 1% 7.609m ± 1% ~ (p=0.184 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.tmpfs-16 7.108m ± 1% 7.088m ± 2% ~ (p=0.838 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.tmpfs-16 6.327m ± 3% 6.330m ± 2% ~ (p=0.539 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.bind-16 549.1µ ± 2% 556.4µ ± 3% ~ (p=0.696 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.bind-16 410.9µ ± 2% 428.5µ ± 5% +4.27% (p=0.039 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.bind-16 557.4µ ± 17% 517.7µ ± 12% ~ (p=0.780 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.bind-16 8.407m ± 1% 8.339m ± 1% ~ (p=0.110 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.bind-16 7.917m ± 2% 7.915m ± 1% ~ (p=0.926 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.bind-16 7.437m ± 2% 7.388m ± 4% ~ (p=0.270 n=16)
geomean 1.923m 1.879m -2.27%
│ before │ after │
│ average_latency.s │ average_latency.s vs base │
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.tmpfs-16 300.0µ ± 33% 300.0µ ± 33% ~ (p=1.000 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.tmpfs-16 8.900m ± 7% 8.850m ± 3% ~ (p=0.744 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.tmpfs-16 97.40m ± 9% 102.25m ± 11% ~ (p=0.327 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.tmpfs-16 7.400m ± 1% 7.300m ± 1% ~ (p=0.277 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.tmpfs-16 468.2m ± 2% 460.6m ± 3% ~ (p=0.193 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.tmpfs-16 3.074 ± 3% 3.087 ± 3% ~ (p=0.309 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.bind-16 300.0µ ± 0% 300.0µ ± 0% ~ (p=1.000 n=16) ¹
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.bind-16 10.10m ± 4% 10.45m ± 5% ~ (p=0.115 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.bind-16 103.4m ± 6% 112.6m ± 7% +8.95% (p=0.039 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.bind-16 8.100m ± 1% 8.050m ± 1% ~ (p=0.132 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.bind-16 525.7m ± 3% 525.7m ± 2% ~ (p=0.985 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.bind-16 3.636 ± 2% 3.628 ± 4% ~ (p=0.318 n=16)
geomean 39.27m 39.69m +1.08%
¹ all samples are equal
│ before │ after │
│ requests_per_second.QPS │ requests_per_second.QPS vs base │
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.tmpfs-16 3.970k ± 1% 3.970k ± 1% ~ (p=0.830 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.tmpfs-16 8.406k ± 6% 8.277k ± 4% ~ (p=0.515 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.tmpfs-16 3.471k ± 61% 5.952k ± 15% ~ (p=0.110 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.tmpfs-16 135.7 ± 1% 136.6 ± 1% ~ (p=0.227 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.tmpfs-16 146.8 ± 1% 146.9 ± 2% ~ (p=0.874 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.tmpfs-16 165.7 ± 2% 165.9 ± 2% ~ (p=0.635 n=16)
NginxDocSize/filesize.100Kb/concurrency.1/filesystem.bind-16 3.711k ± 1% 3.707k ± 1% ~ (p=0.759 n=16)
NginxDocSize/filesize.100Kb/concurrency.100/filesystem.bind-16 7.235k ± 3% 7.025k ± 3% ~ (p=0.146 n=16)
NginxDocSize/filesize.100Kb/concurrency.1000/filesystem.bind-16 3.509k ± 48% 4.331k ± 28% ~ (p=0.724 n=16)
NginxDocSize/filesize.10Mb/concurrency.1/filesystem.bind-16 122.9 ± 1% 124.1 ± 1% ~ (p=0.088 n=16)
NginxDocSize/filesize.10Mb/concurrency.100/filesystem.bind-16 131.0 ± 1% 130.9 ± 2% ~ (p=1.000 n=16)
NginxDocSize/filesize.10Mb/concurrency.1000/filesystem.bind-16 139.8 ± 2% 140.9 ± 2% ~ (p=0.283 n=16)
geomean 811.1 861.9 +6.26%Benchmark environment
GCE c4-standard-4/16/48 (vcpus=), COS cos-129-19506-448-8 (kernel 6.12.105+), hyperdisk-balanced, docker 27.4.1 with cgroup v2, systrap. before is master at 2511bc60218a, after is the patch above. The checkpoint/restore harness checkpoints one alpine container (overlay rootfs plus /etc/hosts and /etc/resolv.conf file binds, as docker does) and per round restores it, timing runsc restore and then sampling the container cgroup; the docker harness starts N idle alpine containers per round (DockerRunAll is the time for all N docker run -d), waits, and samples each container's cgroup. Memory is memory.current minus page cache (file - shmem) and reclaimable slab; arms alternate ABBA per round with caches dropped before each. The //test/benchmarks blocks are the real test binaries at 2511bc60218a on c4-standard-16, one invocation per sample, alternating arms; the last block is an isolated re-run of the nginx cells. The two custom harnesses ran a revision of the patch that differs from the final one only by an unneeded lock in the post-restore path; the //test/benchmarks blocks ran the final patch.
Activity
0No activity yet.