#16171
Optimization: Copy only committed pages when breaking copy-on-write
The idea
When a write breaks copy-on-write on private anonymous memory (typically after fork()), the Sentry copies up to a huge page (2 MiB) around the faulting address by reading the whole range through its internal memfd mapping into a new AllocateAndWritePopulate allocation. Reading a hole in a MAP_SHARED memfd mapping allocates a zero-filled shmem page, and the destination is committed in full. As a result, each break over sparsely-used memory (heaps, arenas, partly written buffers) can commit up to 4 MiB of zero pages, split between the parent's and child's memory, that stay committed until the memory is freed.
The Sentry could instead find the source's committed subranges (mincore() on its own chunk mapping, plus SEEK_DATA for non-resident runs so swapped-out pages still count), allocate the destination uncommitted, and copy only those subranges. This is only safe for private pmas, whose pages can't change during the copy. Fully committed ranges, disk-backed MemoryFiles, and any errors would fall back to today's copy.
Performance impact
- Python processes that fork (fork+exec, a forked worker, a
multiprocessingpool): 0.6-1.0 MiB fewer committed zero pages per sandbox. Children spawned withvfork/posix_spawn(Pythonsubprocess, Nodechild_process) are unaffected. //test/benchmarks/base:usage_testSizeNginx memory -6.5% andhackbenchprocess mode -4.1% (both p<0.01). Startup, checkpoint/restore, size, rubydev, bazel and httpd are unchanged.- Synthetic sparse heaps: COW-break time -30% (worst case: every other page committed) to -90%, committed memory -96%.
- Cost: a break of fully-committed memory pays about 12 µs for the
mincore()scan (vs about 216 µs for the copy), which is +1.9-2.1% in microbenchmarks that do only dense COW breaks.
Beat the autoresearch
The following benchmark results were benchmarked using this patch: https://github.com/google/gvisor/compare/master...EtiennePerot:gvisor:autoresearch/cow-skip-holes
Add value by beating these benchmark numbers, or reducing any regressions they cause, or both
goos: linux
goarch: amd64
cpu: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
pkg: fork/copy-on-write workload harness (docker + runsc, container memory cgroup)
│ before │ after │
│ sec/op │ sec/op vs base │
PyAgentForkExec-16 262.1m ± 5% 255.5m ± 9% ~ (p=0.445 n=16)
PyAgentForkChildGC-16 2.084 ± 0% 2.083 ± 0% ~ (p=0.196 n=16)
PyAgentMultiprocessingForkPool-16 64.58m ± 3% 64.90m ± 3% ~ (p=0.867 n=16)
PyAgentSubprocessVfork-16 33.01m ± 2% 32.40m ± 2% ~ (p=0.138 n=16)
BashSubshells-16 557.2m ± 5% 567.6m ± 7% ~ (p=0.590 n=16)
DenseHeapForkParentWrites-16 47.45m ± 1% 48.44m ± 1% +2.07% (p=0.000 n=16)
NodeExecSync-16 110.5m ± 1% 109.2m ± 13% ~ (p=0.080 n=16)
NodeSpawnSync-16 102.0m ± 1% 101.8m ± 2% ~ (p=0.867 n=16)
SparseHeapForkParentWrites-16 62.472m ± 1% 5.921m ± 3% -90.52% (p=0.000 n=16)
CopyOnWriteBreaks/heap=64MiB/written=1page-per-64k-16 79.42m ± 0% 35.08m ± 0% -55.83% (n=128)
CopyOnWriteBreaks/heap=64MiB/written=1page-per-4k-16 47.82m ± 0% 48.74m ± 0% +1.92% (p=0.000 n=128)
CopyOnWriteBreaks/heap=64MiB/written=1page-per-8k-16 65.18m ± 0% 45.80m ± 0% -29.74% (n=128)
geomean 115.0m 85.77m -25.44%
│ before │ after │
│ nonreclaim-growth-B │ nonreclaim-growth-B vs base │
PyAgentForkExec-16 4.546Mi ± 5% 3.226Mi ± 25% -29.03% (p=0.000 n=16)
PyAgentForkChildGC-16 2.227Mi ± 21% 1.051Mi ± 66% -52.82% (p=0.000 n=16)
PyAgentMultiprocessingForkPool-16 2.587Mi ± 13% 1.931Mi ± 22% -25.37% (p=0.004 n=16)
PyAgentSubprocessVfork-16 2.013Mi ± 13% 1.896Mi ± 17% ~ (p=0.515 n=16)
BashSubshells-16 4.350Mi ± 7% 4.498Mi ± 8% ~ (p=0.402 n=16)
DenseHeapForkParentWrites-16 66.90Mi ± 3% 64.03Mi ± 1% -4.29% (p=0.000 n=16)
NodeExecSync-16 1177.7Ki ± 19% 722.5Ki ± 49% -38.65% (p=0.015 n=16)
NodeSpawnSync-16 1069.3Ki ± 67% 833.4Ki ± 86% ~ (p=0.287 n=16)
SparseHeapForkChildWrites-16 127.227Mi ± 1% 2.964Mi ± 55% -97.67% (p=0.000 n=16)
SparseHeapForkParentWrites-16 125.353Mi ± 0% 4.508Mi ± 30% -96.40% (p=0.000 n=16)
geomean 7.028Mi 2.779Mi -60.46%
pkg: gvisor.dev/gvisor/test/benchmarks/base
│ before │ after │
│ sec/op │ sec/op vs base │
SizeEmpty-16 371.5m ± 1% 372.6m ± 2% ~ (p=0.985 n=16)
SizeNginx-16 752.4m ± 2% 754.1m ± 1% ~ (p=0.642 n=16)
SizeNode-16 874.1m ± 2% 873.1m ± 1% ~ (p=0.696 n=16)
Hackbench/ipcMode.pipe/processMode.thread-16 22.55m ± 6% 22.54m ± 5% ~ (p=0.985 n=16)
Hackbench/ipcMode.socket/processMode.process-16 63.81m ± 1% 61.16m ± 2% -4.14% (p=0.000 n=16)
StartupEmpty-16 185.6m ± 1% 184.6m ± 2% ~ (p=0.491 n=16)
StartupNginx-16 495.9m ± 1% 493.4m ± 1% ~ (p=0.287 n=16)
StartupNode-16 619.9m ± 1% 614.8m ± 1% ~ (p=0.119 n=16)
CheckpointEmpty-16 60.19m ± 1% 60.52m ± 2% ~ (p=0.780 n=16)
RestoreEmpty-16 187.0m ± 2% 186.8m ± 1% ~ (p=0.590 n=16)
CheckpointNginx-16 96.57m ± 2% 96.71m ± 2% ~ (p=0.752 n=16)
RestoreNginx-16 236.5m ± 1% 236.0m ± 2% ~ (p=0.985 n=16)
geomean 203.9m 203.0m -0.44%
│ before │ after │
│ execution_time.s │ execution_time.s vs base │
Hackbench/ipcMode.pipe/processMode.thread-16 53.00m ± 9% 56.50m ± 20% ~ (p=0.889 n=16)
Hackbench/ipcMode.socket/processMode.process-16 105.0m ± 6% 102.0m ± 3% ~ (p=0.155 n=16)
geomean 74.60m 75.91m +1.76%
│ before │ after │
│ execution_time_per_loop.s │ execution_time_per_loop.s vs base │
Hackbench/ipcMode.pipe/processMode.thread-16 5.300m ± 9% 5.650m ± 20% ~ (p=0.889 n=16)
Hackbench/ipcMode.socket/processMode.process-16 10.50m ± 6% 10.20m ± 3% ~ (p=0.155 n=16)
geomean 7.460m 7.591m +1.76%
│ before │ after │
│ average_container_size.bytes │ average_container_size.bytes vs base │
SizeEmpty-16 19.67M ± 10% 21.10M ± 21% ~ (p=0.341 n=16)
SizeNginx-16 31.83M ± 11% 31.32M ± 9% ~ (p=0.956 n=16)
SizeNode-16 56.10M ± 5% 56.41M ± 6% ~ (p=0.515 n=16)
geomean 32.75M 33.41M +2.01%
pkg: gvisor.dev/gvisor/test/benchmarks/base (usage_test)
│ before │ after │
│ sec/op │ sec/op vs base │
SizeEmpty-16 2.097 ± 0% 2.095 ± 0% ~ (p=0.102 n=16)
SizeNginx-16 2.141 ± 0% 2.142 ± 0% ~ (p=0.956 n=16)
SizeNode-16 2.126 ± 0% 2.128 ± 0% ~ (p=0.138 n=16)
geomean 2.121 2.122 +0.02%
│ before │ after │
│ average_container_memory_usage.bytes │ average_container_memory_usage.bytes vs base │
SizeEmpty-16 14.86M ± 1% 14.89M ± 1% ~ (p=0.590 n=16)
SizeNginx-16 23.99M ± 1% 22.42M ± 1% -6.55% (p=0.000 n=16)
SizeNode-16 48.81M ± 0% 48.78M ± 0% ~ (p=0.838 n=16)
geomean 25.91M 25.34M -2.19%
pkg: gvisor.dev/gvisor/test/benchmarks/fs
│ before │ after │
│ sec/op │ sec/op vs base │
RubyNoOpTest/page_cache.clean/filesystem.bindfs-16 662.2m ± 5% 662.8m ± 3% ~ (p=0.491 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.bindfs-16 277.5m ± 4% 271.7m ± 2% ~ (p=0.381 n=16)
RubyNoOpTest/page_cache.clean/filesystem.tmpfs-16 682.9m ± 6% 645.4m ± 3% -5.49% (p=0.043 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.tmpfs-16 271.8m ± 2% 272.7m ± 1% ~ (p=0.515 n=16)
RubyNoOpTest/page_cache.clean/filesystem.rootfs-16 660.5m ± 3% 643.8m ± 2% -2.53% (p=0.035 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.rootfs-16 269.0m ± 4% 272.9m ± 2% ~ (p=0.171 n=16)
BuildABSL/page_cache.clean/filesystem.bindfs-16 37.11 ± 5% 36.22 ± 6% ~ (p=0.515 n=16)
BuildABSL/page_cache.dirty/filesystem.bindfs-16 37.46 ± 7% 36.48 ± 5% ~ (p=0.491 n=16)
BuildABSL/page_cache.clean/filesystem.tmpfs-16 37.29 ± 7% 36.75 ± 4% ~ (p=0.780 n=16)
BuildABSL/page_cache.dirty/filesystem.tmpfs-16 37.58 ± 6% 35.70 ± 7% ~ (p=0.838 n=16)
BuildABSL/page_cache.clean/filesystem.rootfs-16 36.67 ± 5% 37.50 ± 5% ~ (p=0.838 n=16)
BuildABSL/page_cache.dirty/filesystem.rootfs-16 37.58 ± 7% 36.12 ± 4% ~ (p=0.381 n=16)
geomean 3.990 3.918 -1.81%
pkg: gvisor.dev/gvisor/test/benchmarks/network
│ before │ after │
│ sec/op │ sec/op vs base │
Httpd/filesize.10Mb/concurrency.1-16 47.74m ± 1% 47.93m ± 1% ~ (p=0.539 n=16)
Httpd/filesize.10Mb/concurrency.100-16 8.951m ± 2% 9.107m ± 7% ~ (p=0.080 n=16)
Httpd/filesize.1Kb/concurrency.1-16 691.4µ ± 2% 690.5µ ± 1% ~ (p=0.616 n=16)
Httpd/filesize.1Kb/concurrency.100-16 5.450m ± 0% 5.445m ± 0% ~ (p=0.696 n=16)
Httpd/filesize.100Kb/concurrency.1-16 1.119m ± 1% 1.122m ± 1% ~ (p=0.897 n=16)
Httpd/filesize.100Kb/concurrency.100-16 5.479m ± 89% 5.474m ± 1% ~ (p=0.752 n=16)
geomean 4.632m 4.648m +0.34%
│ before │ after │
│ average_latency.s │ average_latency.s vs base │
Httpd/filesize.10Mb/concurrency.1-16 47.40m ± 0% 47.60m ± 1% ~ (p=0.505 n=16)
Httpd/filesize.10Mb/concurrency.100-16 462.3m ± 3% 473.6m ± 8% ~ (p=0.305 n=16)
Httpd/filesize.1Kb/concurrency.1-16 400.0µ ± 0% 400.0µ ± 0% ~ (p=1.000 n=16) ¹
Httpd/filesize.1Kb/concurrency.100-16 46.20m ± 71% 35.40m ± 52% ~ (p=0.200 n=16)
Httpd/filesize.100Kb/concurrency.1-16 800.0µ ± 0% 800.0µ ± 0% ~ (p=1.000 n=16) ¹
Httpd/filesize.100Kb/concurrency.100-16 54.70m ± 65% 87.90m ± 69% ~ (p=0.055 n=16)
geomean 16.15m 16.80m +4.02%
¹ all samples are equal
│ before │ after │
│ requests_per_second.QPS │ requests_per_second.QPS vs base │
Httpd/filesize.10Mb/concurrency.1-16 21.09 ± 0% 21.01 ± 1% ~ (p=0.520 n=16)
Httpd/filesize.10Mb/concurrency.100-16 115.8 ± 2% 113.8 ± 7% ~ (p=0.088 n=16)
Httpd/filesize.1Kb/concurrency.1-16 2.659k ± 1% 2.643k ± 1% ~ (p=0.187 n=16)
Httpd/filesize.1Kb/concurrency.100-16 194.8 ± 0% 194.9 ± 0% ~ (p=0.334 n=16)
Httpd/filesize.100Kb/concurrency.1-16 1.235k ± 1% 1.244k ± 1% ~ (p=0.249 n=16)
Httpd/filesize.100Kb/concurrency.100-16 193.6 ± 0% 193.9 ± 0% ~ (p=0.845 n=16)
geomean 259.1 258.3 -0.31%Benchmark environment
GCE c4-standard-16, COS cos-129-19506-448-8 (kernel 6.12.105+), hyperdisk-balanced, shmem THP never, no swap, docker with runtimes for both builds (default flags, systrap). before is master at 2511bc60218a, after is the patch above. Interleaved runs (ABBA by round).
- The fork/COW harness runs each workload in a fresh container. Times are measured inside the workload, and
nonreclaim-growth-Bis the growth of the container cgroup's non-reclaimable memory (memory.currentminus page cache other than shmem, minus reclaimable slab) across the fork phase. CopyOnWriteBreakstimes 32 parent writes that break COW over a 64 MiB mapping with one written page per stride. Its 8 samples per container run are not independent.- The
//test/benchmarksrows are the upstream binaries. Bazel was pooled over two identical VMs. The httpd concurrency-100 rows are bimodal in both builds.
Activity
0No activity yet.