Optimization: Keep unused /proc and /sys trees out of checkpoints

Open

The idea

gVisor builds all of /proc/sys (about 100 inodes per procfs mount) and all of /sys when the filesystems are mounted, including 6 inodes per visible CPU under /sys/devices/system/cpu/cpuN/topology, and it stores /proc/cpuinfo as a ~1 KB-per-CPU string. Checkpoints also include every /proc/[pid] tree (about 40 inodes each) that is only kept alive by the procfs dentry cache. Workloads rarely touch most of this, but every checkpoint encodes it all, and every restore decodes it all too, so on a large host, a small sandbox's state file is dominated by synthetic filesystem objects that are trivially reconstructible.

gVisor could instead:

Create the /proc/sys directories and the /sys/devices/system/cpu and cpuN directories on first lookup, open, getdents or stat call.
Unpopulated directories can be saved empty, and populated on demand after restore.
/proc/cpuinfo can be generated on read.
procfs and sysfs cached dentries should be evicted in PrepareSave, so dentry-cache-only /proc/[pid] trees are recreated on demand.

The visible tree stays identical (names, modes, link counts, contents, rmdir errors). The only visible difference: after a cross-host restore, the made-up amd64 cpu MHz/bogomips values in /proc/cpuinfo come from the current host. If that's a dealbreaker, then we should save only those values in the checkpoint metadata.

Performance impact

Idle sandbox with 96 visible CPUs: checkpoint CPU -53%, state file 373 → 114 KiB, restore CPU -14%, restored Sentry anonymous memory -13%.
A sandbox that has run ps at least once: checkpoint CPU -28% to -49% and restore CPU -3% to -14% at 2 to 96 visible CPUs; idle sandboxes with 4-16 visible CPUs use 6-15% less checkpoint CPU.
Even with every /proc/sys and /sys entry fully-populated, the state file is 41% smaller and restore CPU 11% lower.
The /proc/cpuinfo part only shrinks the state file (~1 KiB per CPU); at 96 CPUs it measured ~1 ms slower Kernel save than the rest of the change alone (p≈0.03), possibly a GC pacing effect.
//test/benchmarks/base: CheckpointEmpty -5.8%, per-container memory (usage_test) -1.4% to -2.7%, startup and size_test unchanged.

Beat the autoresearch

The following benchmark results were benchmarked using this patch: https://github.com/google/gvisor/compare/master...EtiennePerot:gvisor:autoresearch/lazy-kernfs-save

Add value by beating these benchmark numbers, or reducing any regressions they cause, or both

pkg: gvisor.dev/gvisor/test/benchmarks/base (startup_test)
                   │    before    │               after                │
                   │    sec/op    │   sec/op     vs base               │
StartupEmpty-16       186.7m ± 2%   188.8m ± 2%       ~ (p=0.341 n=16)
StartupNginx-16       503.6m ± 2%   506.0m ± 1%       ~ (p=0.210 n=16)
StartupNode-16        623.0m ± 1%   622.2m ± 1%       ~ (p=0.696 n=16)
CheckpointEmpty-16    65.13m ± 3%   61.38m ± 3%  -5.75% (p=0.000 n=16)
RestoreEmpty-16       190.3m ± 2%   189.6m ± 1%       ~ (p=0.491 n=16)
CheckpointNginx-16   102.18m ± 2%   99.67m ± 2%  -2.46% (p=0.023 n=16)
RestoreNginx-16       243.4m ± 2%   240.9m ± 2%  -1.04% (p=0.039 n=16)
geomean               210.1m        207.6m       -1.19%

             │                before                │                            after                            │
             │ average_container_memory_usage.bytes │ average_container_memory_usage.bytes  vs base               │
SizeNginx-16                            23.99M ± 1%                            23.65M ± 1%  -1.41% (p=0.000 n=16)
SizeNode-16                             49.47M ± 1%                            49.07M ± 2%       ~ (p=0.468 n=16)
SizeEmpty-16                            14.92M ± 2%                            14.51M ± 2%  -2.74% (p=0.000 n=16)
geomean                                 26.07M                                 25.63M       -1.65%

pkg: runsc checkpoint/restore harness (alpine, runsc run/checkpoint/restore)
                                │    before    │                after                 │
                                │    sec/op    │    sec/op     vs base                │
Checkpoint/cpus=96/idle           31.53m ±  9%   20.35m ±  7%  -35.46% (p=0.000 n=16)
Restore/cpus=96/idle              52.24m ±  5%   44.73m ±  3%  -14.37% (p=0.000 n=16)
Start/cpus=96/idle                355.5m ±  3%   340.2m ±  8%   -4.30% (p=0.007 n=16)
Checkpoint/cpus=96/ps             38.09m ±  5%   26.40m ±  8%  -30.71% (p=0.000 n=16)
Restore/cpus=96/ps                75.85m ±  9%   65.20m ±  6%  -14.03% (p=0.000 n=16)
Start/cpus=96/ps                  348.7m ±  8%   330.4m ±  2%        ~ (p=0.073 n=16)
Checkpoint/cpus=96/walk-all       33.36m ±  6%   29.75m ±  3%  -10.82% (p=0.000 n=16)
Restore/cpus=96/walk-all          56.48m ±  6%   50.46m ±  4%  -10.65% (p=0.001 n=16)
Start/cpus=96/walk-all            327.0m ±  5%   333.2m ±  5%        ~ (p=0.616 n=16)
Checkpoint/cpus=16/idle           22.83m ±  8%   21.24m ±  8%   -6.96% (p=0.035 n=16)
Restore/cpus=16/idle              45.06m ±  6%   43.67m ±  4%        ~ (p=0.361 n=16)
Start/cpus=16/idle                340.7m ±  5%   343.7m ±  3%        ~ (p=0.926 n=16)
Checkpoint/cpus=16/ps             34.31m ±  7%   23.33m ±  7%  -31.99% (p=0.000 n=16)
Restore/cpus=16/ps                68.27m ±  6%   62.65m ±  9%        ~ (p=0.128 n=16)
Start/cpus=16/ps                  347.1m ±  4%   334.5m ±  6%   -3.64% (p=0.003 n=16)
Checkpoint/cpus=4/idle            19.59m ±  9%   20.69m ±  9%        ~ (p=0.402 n=16)
Restore/cpus=4/idle               44.87m ±  7%   44.05m ±  5%        ~ (p=0.724 n=16)
Start/cpus=4/idle                 337.5m ±  5%   329.7m ±  5%        ~ (p=0.224 n=16)
Checkpoint/cpus=4/ps              31.74m ±  7%   22.49m ±  7%  -29.15% (p=0.000 n=16)
Restore/cpus=4/ps                 63.51m ±  6%   63.46m ± 11%        ~ (p=0.780 n=16)
Start/cpus=4/ps                   331.3m ±  4%   329.7m ±  6%        ~ (p=0.669 n=16)
Checkpoint/cpus=2,quota=0.25/ps   35.82m ± 30%   22.61m ±  8%  -36.87% (p=0.000 n=16)
Restore/cpus=2,quota=0.25/ps      377.8m ±  7%   365.6m ±  6%        ~ (p=0.491 n=16)
Start/cpus=2,quota=0.25/ps        476.6m ±  6%   485.9m ±  8%        ~ (p=0.926 n=16)
geomean                           91.92m         81.69m        -11.13%

                                │    before     │                after                 │
                                │  cpu-sec/op   │  cpu-sec/op   vs base                │
Checkpoint/cpus=96/idle            47.90m ±  4%   22.45m ±  8%  -53.14% (p=0.000 n=16)
Restore/cpus=96/idle               99.71m ±  5%   85.62m ±  5%  -14.13% (p=0.000 n=16)
Start/cpus=96/idle                 94.07m ±  4%   90.34m ±  5%   -3.97% (p=0.003 n=16)
Checkpoint/cpus=96/ps              62.19m ± 19%   45.08m ± 11%  -27.52% (p=0.000 n=16)
Restore/cpus=96/ps                 124.6m ±  6%   107.1m ±  3%  -14.02% (p=0.000 n=16)
Start/cpus=96/ps                   91.45m ±  2%   90.23m ±  3%        ~ (p=0.210 n=16)
Checkpoint/cpus=96/walk-all        49.60m ±  8%   47.51m ± 13%        ~ (p=0.323 n=16)
Restore/cpus=96/walk-all          108.89m ±  6%   96.85m ±  2%  -11.06% (p=0.000 n=16)
Start/cpus=96/walk-all             93.08m ±  5%   90.33m ±  2%        ~ (p=0.224 n=16)
Checkpoint/cpus=16/idle            14.98m ±  3%   12.71m ±  3%  -15.17% (p=0.000 n=16)
Restore/cpus=16/idle               74.21m ±  4%   73.83m ±  2%        ~ (p=0.642 n=16)
Start/cpus=16/idle                 82.98m ±  3%   83.33m ±  3%        ~ (p=0.423 n=16)
Checkpoint/cpus=16/ps              35.58m ±  3%   18.45m ± 23%  -48.15% (p=0.000 n=16)
Restore/cpus=16/ps                 99.11m ±  4%   91.48m ±  3%   -7.70% (p=0.000 n=16)
Start/cpus=16/ps                   82.21m ±  2%   82.08m ±  5%        ~ (p=0.323 n=16)
Checkpoint/cpus=4/idle             12.67m ±  4%   11.85m ±  7%   -6.51% (p=0.011 n=16)
Restore/cpus=4/idle                72.12m ±  2%   70.43m ±  2%        ~ (p=0.149 n=16)
Start/cpus=4/idle                  80.09m ±  4%   79.54m ±  4%        ~ (p=0.838 n=16)
Checkpoint/cpus=4/ps               33.33m ±  3%   17.15m ±  3%  -48.55% (p=0.000 n=16)
Restore/cpus=4/ps                  94.10m ±  4%   86.62m ±  1%   -7.95% (p=0.000 n=16)
Start/cpus=4/ps                    80.83m ±  3%   81.51m ±  3%        ~ (p=0.752 n=16)
Checkpoint/cpus=2,quota=0.25/ps    26.44m ± 16%   16.26m ±  4%  -38.50% (p=0.000 n=16)
Restore/cpus=2,quota=0.25/ps       93.57m ±  3%   90.88m ±  4%   -2.87% (p=0.039 n=16)
Start/cpus=2,quota=0.25/ps         77.74m ±  3%   77.56m ±  7%        ~ (p=0.642 n=16)
geomean                            63.00m         53.58m        -14.95%

                                │    before    │                after                 │
                                │   state-B    │   state-B     vs base                │
Checkpoint/cpus=96/idle           372.5Ki ± 0%   113.8Ki ± 0%  -69.46% (p=0.000 n=16)
Checkpoint/cpus=96/ps             611.3Ki ± 0%   272.6Ki ± 0%  -55.41% (p=0.000 n=16)
Checkpoint/cpus=96/walk-all       464.5Ki ± 0%   274.5Ki ± 0%  -40.91% (p=0.000 n=16)
Checkpoint/cpus=16/idle           173.9Ki ± 0%   113.3Ki ± 0%  -34.87% (p=0.000 n=16)
Checkpoint/cpus=16/ps             412.6Ki ± 0%   252.8Ki ± 0%  -38.74% (p=0.000 n=16)
Checkpoint/cpus=4/idle            145.4Ki ± 0%   113.2Ki ± 0%  -22.14% (p=0.000 n=16)
Checkpoint/cpus=4/ps              384.0Ki ± 0%   249.8Ki ± 0%  -34.96% (p=0.000 n=16)
Checkpoint/cpus=2,quota=0.25/ps   379.4Ki ± 0%   249.3Ki ± 0%  -34.28% (p=0.000 n=16)
geomean                           335.6Ki        190.3Ki       -43.30%

                             │    before     │                 after                 │
                             │ sentry-anon-B │ sentry-anon-B  vs base                │
Restore/cpus=96/idle            14.69Mi ± 2%    12.79Mi ± 2%  -12.90% (p=0.000 n=16)
Start/cpus=96/idle              11.82Mi ± 3%    11.46Mi ± 2%   -3.09% (p=0.011 n=16)
Restore/cpus=96/ps              16.67Mi ± 2%    14.88Mi ± 2%  -10.69% (p=0.000 n=16)
Start/cpus=96/ps                15.24Mi ± 4%    14.40Mi ± 6%   -5.55% (p=0.002 n=16)
Restore/cpus=96/walk-all        15.40Mi ± 3%    14.82Mi ± 1%   -3.80% (p=0.000 n=16)
Start/cpus=96/walk-all          13.94Mi ± 4%    13.64Mi ± 3%        ~ (p=0.088 n=16)
Restore/cpus=16/idle            11.90Mi ± 2%    11.55Mi ± 2%   -2.95% (p=0.005 n=16)
Start/cpus=16/idle              10.76Mi ± 2%    10.83Mi ± 4%        ~ (p=0.773 n=16)
Restore/cpus=16/ps              14.48Mi ± 1%    13.31Mi ± 2%   -8.03% (p=0.000 n=16)
Start/cpus=16/ps                13.54Mi ± 9%    13.20Mi ± 9%        ~ (p=0.126 n=16)
Restore/cpus=4/idle             11.38Mi ± 2%    11.17Mi ± 3%   -1.77% (p=0.024 n=16)
Start/cpus=4/idle               10.92Mi ± 3%    10.74Mi ± 3%   -1.63% (p=0.017 n=16)
Restore/cpus=4/ps               13.63Mi ± 2%    12.86Mi ± 2%   -5.69% (p=0.000 n=16)
Start/cpus=4/ps                 13.06Mi ± 2%    12.76Mi ± 1%        ~ (p=0.053 n=16)
Restore/cpus=2,quota=0.25/ps    13.85Mi ± 2%    12.78Mi ± 4%   -7.73% (p=0.000 n=16)
Start/cpus=2,quota=0.25/ps      12.82Mi ± 3%    12.25Mi ± 2%   -4.42% (p=0.026 n=16)
geomean                         13.28Mi         12.65Mi        -4.73%

Benchmark environment

//test/benchmarks/base ran on a GCE c4-standard-16 and the checkpoint/restore harness on a c4-standard-96, both COS cos-129-19506-448-8 (kernel 6.12.105+, cgroup v2), hyperdisk-balanced. before is master at 2511bc60218a, after is the patch above; arms interleaved, n=16 per arm. The //test/benchmarks/base binaries ran against two docker runtimes (size_test's SizeEmpty fails on both arms because of a harness bug, so it is absent). The harness runs an alpine container with runsc run, waits 1.5 s, then times runsc checkpoint and runsc restore (sec/op), and records the sandbox cgroup's CPU usage for each step (cpu-sec/op), the checkpoint.img size (state-B) and the Sentry's RssAnon 1.5 s after start/restore (sentry-anon-B). idle runs sleep; ps reads a sysctl and /sys/devices/system/cpu/online and runs ps before and after starting 8 sleepers; walk-all stats every file under /proc/sys and /sys first. cpus=N is the number of CPUs the sandbox sees (--cpu-num-from-quota with a CPU quota of N cores; cpus=96 is uncapped; quota=0.25 gives 2).

Activity

No activity yet.