Optimization: Start the gofer with a GOMAXPROCS bounded by its cgroup and capped at 8

Open

The idea

runsc strips any inherited GOMAXPROCS from the gofer's environment, so the gofer's Go runtime sizes itself from the host CPU count, or from cpu.max of its leaf cgroup only. It misses CPU limits set on a parent cgroup (systemd slices, Kubernetes pod-level limits, runsc's own cgroupfs layout) and uses every host CPU when there is no limit at all, which is the Docker and Burstable/BestEffort default. The gofer spends nearly all of its time in host syscalls and gains little from many Ps, but each P costs heap (the P itself plus its mcache) that the runtime never frees; above roughly 160 Ps the P structs alone trigger a GC during init, which starts per-P GC worker threads.

runsc could put GOMAXPROCS=min(NumCPU, cgroup CPU count, 8) in the gofer's environment, where the cgroup CPU count is the one the gofer is cloned into, computed with the same rule the Sentry uses (calculateCPUNum: cpuset, quota with the cgroup v2 parent fallback, floor of 2). It has to be an environment variable because Ps are allocated at schedinit, before the gofer could call runtime.GOMAXPROCS. If runsc also starts restoring the runtime's default GOMAXPROCS in the gofer (for example after bounding GOMAXPROCS during runsc's own init on large hosts), this bound must take precedence. Forcing a GC or FreeOSMemory in the idle gofer to shrink it instead makes it larger, because it starts the GC workers that an idle gofer never otherwise needs.

Note: This is complementary with #16140, which sizes runsc generally.

Performance impact

Gofer PSS with no CPU limit: 3.89 → 3.15 MiB at 48 vCPUs, 4.77 → 3.16 MiB at 96, 8.81 → 3.15 MiB at 192 (and 54 → 7 threads at 192).
With the CPU limit on a parent cgroup, the saving starts lower: -0.4 MiB per gofer at 16 vCPUs, -1.8 MiB at 96, -6.0 MiB at 192. No change at 2 vCPUs or with the quota on the leaf cgroup, where Go already gets it right.
docker run startup and gofer-bound file IO (directfs off) are unchanged from 2 to 192 vCPUs. 64-job file creation looks about 1.4% slower at 96 and 192 vCPUs, not significant (p=0.09/0.10).
A cap of 2 would save another ~240 KiB per gofer but makes a cold-cache parallel stat storm 36% slower; with 8 it matches master.
//test/benchmarks (base startup/size/usage, fs rubydev/bazel/fio): no change at p<0.01.

Beat the autoresearch

The following benchmark results were benchmarked using this patch: https://github.com/google/gvisor/compare/master...EtiennePerot:gvisor:autoresearch/gofer-go-runtime

Add value by beating these benchmark numbers, or reducing any regressions they cause, or both

goos: linux
goarch: amd64
pkg: docker run startup harness
                    │   before    │               after                │
                    │   sec/op    │   sec/op     vs base               │
DockerRun/vcpus=2     283.8m ± 1%   280.5m ± 3%       ~ (p=0.623 n=30)
DockerRun/vcpus=16    296.5m ± 2%   294.1m ± 2%       ~ (p=0.366 n=30)
DockerRun/vcpus=48    304.9m ± 1%   304.2m ± 3%       ~ (p=0.440 n=30)
DockerRun/vcpus=96    328.5m ± 1%   329.0m ± 1%       ~ (p=0.832 n=30)
DockerRun/vcpus=192   382.1m ± 2%   382.8m ± 3%       ~ (p=0.764 n=30)
geomean               317.4m        316.2m       -0.37%

pkg: gofer file harness (directfs off, tmpfs bind mount; cold-cache stat on disk)
                          │   before   │               after               │
                          │   sec/op   │   sec/op    vs base               │
ColdStat/vcpus=96/par=64    1.240 ± 1%   1.240 ± 1%       ~ (p=0.105 n=24)
ColdStat/vcpus=192/par=64   1.640 ± 3%   1.600 ± 3%       ~ (p=0.150 n=16)
geomean                     1.426        1.409       -1.23%

                                 │    before    │               after                │
                                 │     iops     │    iops      vs base               │
Fio/vcpus=16/jobs=16/filecreate    13.70k ±  0%   13.73k ± 0%       ~ (p=0.468 n=16)
Fio/vcpus=16/jobs=16/filestat      22.91k ±  0%   22.92k ± 0%       ~ (p=0.616 n=16)
Fio/vcpus=16/jobs=16/randread      460.5k ±  3%   453.8k ± 4%       ~ (p=0.724 n=16)
Fio/vcpus=16/jobs=16/randwrite     458.7k ±  6%   448.3k ± 4%       ~ (p=0.780 n=16)
Fio/vcpus=96/jobs=16/filestat      13.08k ±  2%   13.13k ± 2%       ~ (p=0.818 n=6)
Fio/vcpus=96/jobs=16/randread      322.3k ±  1%   326.4k ± 4%       ~ (p=0.310 n=6)
Fio/vcpus=96/jobs=16/randwrite     323.6k ±  3%   325.9k ± 3%       ~ (p=0.589 n=6)
Fio/vcpus=96/jobs=64/filecreate    10.49k ±  1%   10.35k ± 1%       ~ (p=0.093 n=6)
Fio/vcpus=96/jobs=64/filestat      17.47k ±  0%   17.51k ± 1%       ~ (p=0.394 n=6)
Fio/vcpus=96/jobs=64/randread      830.9k ±  1%   832.1k ± 1%       ~ (p=0.818 n=6)
Fio/vcpus=96/jobs=64/randwrite     832.7k ±  2%   836.5k ± 2%       ~ (p=0.589 n=6)
Fio/vcpus=192/jobs=16/filecreate   6.620k ±  3%   6.527k ± 8%       ~ (p=0.937 n=6)
Fio/vcpus=192/jobs=16/filestat     10.93k ±  6%   10.94k ± 6%       ~ (p=0.699 n=6)
Fio/vcpus=192/jobs=16/randread     289.1k ±  8%   296.9k ± 7%       ~ (p=0.485 n=6)
Fio/vcpus=192/jobs=16/randwrite    284.5k ± 16%   274.0k ± 7%       ~ (p=0.485 n=6)
Fio/vcpus=192/jobs=64/filecreate   7.804k ±  4%   7.685k ± 2%       ~ (p=0.102 n=16)
Fio/vcpus=192/jobs=64/filestat     12.38k ±  1%   12.49k ± 2%       ~ (p=0.491 n=16)
geomean                            65.00k         64.81k       -0.28%

pkg: gofer memory harness (10 alpine containers per arm per round)
                                │    before    │                after                 │
                                │  Pss_Anon-B  │  Pss_Anon-B   vs base                │
GoferMem/vcpus=2/noquota          1.906Mi ± 1%   1.918Mi ± 1%        ~ (p=0.145 n=60)
GoferMem/vcpus=2/quota=parent     1.908Mi ± 1%   1.916Mi ± 1%        ~ (p=0.249 n=40)
GoferMem/vcpus=16/noquota         2.322Mi ± 1%   2.201Mi ± 3%   -5.21% (p=0.000 n=60)
GoferMem/vcpus=16/quota=parent    2.312Mi ± 2%   1.922Mi ± 1%  -16.89% (p=0.000 n=40)
GoferMem/vcpus=16/quota=leaf      1.920Mi ± 1%   1.934Mi ± 1%        ~ (p=0.233 n=20)
GoferMem/vcpus=48/noquota         2.936Mi ± 1%   2.156Mi ± 1%  -26.55% (n=60)
GoferMem/vcpus=48/quota=parent    2.949Mi ± 2%   1.918Mi ± 0%  -34.97% (p=0.000 n=40)
GoferMem/vcpus=96/noquota         3.803Mi ± 1%   2.172Mi ± 2%  -42.89% (n=60)
GoferMem/vcpus=96/quota=parent    3.758Mi ± 2%   1.920Mi ± 1%  -48.91% (p=0.000 n=40)
GoferMem/vcpus=192/noquota        7.832Mi ± 0%   2.150Mi ± 2%  -72.54% (n=60)
GoferMem/vcpus=192/quota=parent   7.936Mi ± 2%   1.930Mi ± 1%  -75.68% (p=0.000 n=40)
geomean                           3.136Mi        2.009Mi       -35.94%

                                │    before    │                after                 │
                                │    Pss-B     │    Pss-B      vs base                │
GoferMem/vcpus=2/noquota          2.877Mi ± 1%   2.879Mi ± 1%        ~ (p=0.669 n=60)
GoferMem/vcpus=2/quota=parent     2.889Mi ± 1%   2.894Mi ± 1%        ~ (p=0.810 n=40)
GoferMem/vcpus=16/noquota         3.327Mi ± 2%   3.152Mi ± 2%   -5.25% (p=0.000 n=60)
GoferMem/vcpus=16/quota=parent    3.271Mi ± 2%   2.886Mi ± 1%  -11.76% (p=0.000 n=40)
GoferMem/vcpus=16/quota=leaf      2.905Mi ± 2%   2.917Mi ± 1%        ~ (p=0.635 n=20)
GoferMem/vcpus=48/noquota         3.892Mi ± 1%   3.147Mi ± 1%  -19.13% (n=60)
GoferMem/vcpus=48/quota=parent    3.928Mi ± 1%   2.889Mi ± 1%  -26.46% (p=0.000 n=40)
GoferMem/vcpus=96/noquota         4.766Mi ± 1%   3.155Mi ± 2%  -33.81% (n=60)
GoferMem/vcpus=96/quota=parent    4.721Mi ± 2%   2.900Mi ± 1%  -38.58% (p=0.000 n=40)
GoferMem/vcpus=192/noquota        8.806Mi ± 1%   3.150Mi ± 2%  -64.23% (n=60)
GoferMem/vcpus=192/quota=parent   8.882Mi ± 2%   2.897Mi ± 1%  -67.38% (p=0.000 n=40)
geomean                           4.194Mi        2.985Mi       -28.82%

                                │   before    │               after                │
                                │   threads   │  threads    vs base                │
GoferMem/vcpus=2/noquota           7.000 ± 0%   7.000 ± 0%        ~ (p=0.408 n=60)
GoferMem/vcpus=2/quota=parent      7.000 ± 0%   7.000 ± 0%        ~ (p=0.569 n=40)
GoferMem/vcpus=16/noquota          7.000 ± 0%   7.000 ± 0%        ~ (p=0.437 n=60)
GoferMem/vcpus=16/quota=parent     7.000 ± 0%   7.000 ± 0%    0.00% (p=0.027 n=40)
GoferMem/vcpus=16/quota=leaf       7.000 ± 0%   7.000 ± 0%        ~ (p=1.000 n=20)
GoferMem/vcpus=48/noquota          7.000 ± 0%   7.000 ± 0%        ~ (p=1.000 n=60)
GoferMem/vcpus=48/quota=parent     7.000 ± 0%   7.000 ± 0%    0.00% (p=0.000 n=40)
GoferMem/vcpus=96/noquota          7.000 ± 0%   7.000 ± 0%        ~ (p=0.689 n=60)
GoferMem/vcpus=96/quota=parent     7.000 ± 0%   7.000 ± 0%        ~ (p=0.732 n=40)
GoferMem/vcpus=192/noquota        54.000 ± 2%   7.000 ± 0%  -87.04% (n=60)
GoferMem/vcpus=192/quota=parent   53.000 ± 2%   7.000 ± 0%  -86.79% (p=0.000 n=40)
geomean                            10.13        7.000       -30.91%

pkg: gvisor.dev/gvisor/test/benchmarks/base
                │   before    │               after                │
                │   sec/op    │   sec/op     vs base               │
StartupEmpty-16   173.0m ± 3%   174.2m ± 3%       ~ (p=0.361 n=16)
StartupNginx-16   463.8m ± 1%   467.0m ± 1%       ~ (p=0.323 n=16)
StartupNode-16    568.4m ± 1%   564.3m ± 2%       ~ (p=0.539 n=16)
SizeEmpty-16      352.2m ± 2%   348.2m ± 2%       ~ (p=0.254 n=16)
SizeNginx-16      708.1m ± 2%   701.0m ± 1%       ~ (p=0.160 n=16)
SizeNode-16       811.6m ± 1%   804.5m ± 2%       ~ (p=0.149 n=16)
geomean           458.0m        456.2m       -0.39%

             │            before            │                        after                        │
             │ average_container_size.bytes │ average_container_size.bytes  vs base               │
SizeEmpty-16                   21.36M ± 24%                   20.44M ± 13%       ~ (p=0.669 n=16)
SizeNginx-16                   32.12M ±  8%                   34.66M ± 10%       ~ (p=0.287 n=16)
SizeNode-16                    57.26M ±  4%                   56.61M ±  3%       ~ (p=0.539 n=16)
geomean                        33.99M                         34.23M        +0.69%

pkg: gvisor.dev/gvisor/test/benchmarks/base (usage_test)
             │   before   │               after               │
             │   sec/op   │   sec/op    vs base               │
SizeEmpty-16   2.092 ± 0%   2.096 ± 0%  +0.18% (p=0.017 n=16)
SizeNginx-16   2.131 ± 0%   2.138 ± 1%       ~ (p=0.468 n=16)
SizeNode-16    2.103 ± 0%   2.106 ± 0%       ~ (p=0.361 n=16)
geomean        2.109        2.113       +0.21%

             │                before                │                            after                            │
             │ average_container_memory_usage.bytes │ average_container_memory_usage.bytes  vs base               │
SizeEmpty-16                            14.96M ± 1%                            14.79M ± 1%       ~ (p=0.102 n=16)
SizeNginx-16                            23.99M ± 1%                            23.85M ± 1%  -0.60% (p=0.014 n=16)
SizeNode-16                             48.83M ± 0%                            48.79M ± 1%       ~ (p=0.341 n=16)
geomean                                 25.98M                                 25.82M       -0.60%

pkg: gvisor.dev/gvisor/test/benchmarks/fs
                                                                                                        │   before    │               after                │
                                                                                                        │   sec/op    │   sec/op     vs base               │
RubyNoOpTest/page_cache.clean/filesystem.bindfs-16                                                        637.1m ± 2%   635.5m ± 2%       ~ (p=0.897 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.bindfs-16                                                        255.3m ± 2%   251.5m ± 3%       ~ (p=0.061 n=16)
RubyNoOpTest/page_cache.clean/filesystem.tmpfs-16                                                         631.2m ± 2%   626.3m ± 1%       ~ (p=0.323 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.tmpfs-16                                                         255.7m ± 3%   254.5m ± 5%       ~ (p=0.669 n=16)
RubyNoOpTest/page_cache.clean/filesystem.rootfs-16                                                        632.6m ± 3%   624.1m ± 1%       ~ (p=0.128 n=16)
RubyNoOpTest/page_cache.dirty/filesystem.rootfs-16                                                        252.7m ± 3%   249.7m ± 2%       ~ (p=0.445 n=16)
BuildABSL/page_cache.clean/filesystem.bindfs-16                                                            32.15 ± 2%    32.25 ± 1%       ~ (p=0.515 n=16)
BuildABSL/page_cache.dirty/filesystem.bindfs-16                                                            31.90 ± 1%    32.13 ± 2%       ~ (p=0.616 n=16)
BuildABSL/page_cache.clean/filesystem.tmpfs-16                                                             32.08 ± 1%    32.45 ± 4%       ~ (p=0.341 n=16)
BuildABSL/page_cache.dirty/filesystem.tmpfs-16                                                             31.82 ± 2%    31.92 ± 2%       ~ (p=0.926 n=16)
BuildABSL/page_cache.clean/filesystem.rootfs-16                                                            32.28 ± 1%    31.96 ± 2%       ~ (p=0.128 n=16)
BuildABSL/page_cache.dirty/filesystem.rootfs-16                                                            31.83 ± 2%    31.74 ± 3%       ~ (p=0.590 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16            2.889m ± 2%   2.916m ± 2%       ~ (p=0.897 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16             2.952m ± 2%   2.940m ± 1%       ~ (p=0.381 n=16)
FioWrite/operation.write/ioEngine.sync/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16            3.077m ± 1%   3.081m ± 1%       ~ (p=0.809 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.bindfs-16            5.766m ± 2%   5.793m ± 1%       ~ (p=0.305 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.tmpfs-16             4.668m ± 1%   4.676m ± 2%       ~ (p=0.539 n=16)
FioRead/operation.read/ioEngine.libaio/jobs.1/blockSize.4K/directIO.false/filesystem.rootfs-16            5.735m ± 1%   5.768m ± 2%       ~ (p=0.696 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16     31.28m ± 1%   31.32m ± 1%       ~ (p=0.515 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16      13.11m ± 1%   13.09m ± 1%       ~ (p=0.423 n=16)
FioRandRead/operation.randread/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16     13.98m ± 1%   13.88m ± 1%       ~ (p=0.539 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.bindfs-16   14.29m ± 2%   14.25m ± 1%       ~ (p=0.669 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.tmpfs-16    14.26m ± 2%   14.22m ± 1%       ~ (p=0.867 n=16)
FioRandWrite/operation.randwrite/ioEngine.libaio/jobs.8/blockSize.4K/directIO.true/filesystem.rootfs-16   13.73m ± 2%   13.75m ± 1%       ~ (p=0.985 n=16)
geomean                                                                                                   169.0m        168.7m       -0.15%

Benchmark environment

GCE c4-standard-2/16/48/96/192 (vcpus=), COS cos-129-19506-448-8 (kernel 6.12.105+), hyperdisk-balanced, Docker 27.4.1 with the systemd cgroup driver (cgroup v2), default runsc flags. before is master at 2511bc60218a, after is the patch above, arms interleaved on the same VM. The gofer memory harness starts 10 alpine sleep containers per arm per round and reads each gofer's smaps_rollup 10 s later; each sample is one gofer, and the 10 gofers of a round start together, so samples within a round are not fully independent; quota=parent puts a 0.25-CPU quota on a parent slice (--cgroup-parent), quota=leaf uses docker run --cpus=0.25. The startup harness times docker run --rm alpine true. The gofer file harness runs fio with directfs off on a tmpfs bind mount (randread/randwrite go through donated host FDs and serve as controls), and ColdStat times 64 parallel workers each stating 300 files on disk after dropping caches. The test/benchmarks blocks are the upstream benchmark binaries on a c4-standard-16 (so the gofer goes from 16 to 8 Ps there), one invocation per sample.

Activity

No activity yet.