Optimization: Trim three avoidable allocations from the Sentry's boot heap

Open

The idea

Every Sentry carries a few hundred KiB of anonymous memory that it never needs. Three sources are easy to remove. (1) The Go runtime samples heap allocations for profiling by default, and every sampled call site keeps a profiling bucket for the life of the process. Heap profiles can only be collected with --profile, though. (2) Each netstack neighbor cache (ARP and NDP, so two per NIC) is created with a map presized for NeighborCacheSize (512) entries. With Go's swiss maps that is about 41 KiB per cache, even though most caches only ever hold a few entries. (3) The /unimplemented_syscalls metric builds a map from syscall number to a one-element []*metric.FieldValue at boot, so that incrementing it does not allocate.

These are three independent savings:

runsc could add memprofilerate=0 to the Sentry's GODEBUG when --profile is not set. Behaviour with --profile stays the same.
netstack could create neighbor caches without a size hint, since newEntry already enforces the upper bound.
The syscall metric could reuse its existing allowedValues slice and pass the length-1 subslice [sysno:sysno+1], which still doesn't allocate.

Performance impact

Sentry anonymous memory (Pss_Anon) per sandbox: -3.4% to -4.7% (about 340-440 KiB) after boot and after restore. With docker bridge networking, cgroup anonymous memory drops by about 300-400 KiB, including under 1-CPU and 0.25-CPU quotas.
Per trim: the syscall metric saves about 260 KiB, and skipping profiling sampling saves 80 KiB at boot and 230 KiB after restore. The neighbor cache saves an expected ~80 KiB per NIC, but that was not statistically resolved.
//test/benchmarks/base usage_test memory: -3.0% (Empty), -2.4% (Nginx), -1.2% (Node, p=0.04). No other base or network benchmark moves, and nothing gets worse at p<0.01.
Start time and CPU show no significant change.

Beat the autoresearch

The following benchmark results were benchmarked using this patch: https://github.com/google/gvisor/compare/master...EtiennePerot:gvisor:autoresearch/sentry-boot-heap-trims

Add value by beating these benchmark numbers, or reducing any regressions they cause, or both

goos: linux
goarch: amd64
cpu: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz
pkg: boot/restore memory harness (runsc run, --network=none, 30 sandboxes per arm per round)
                              │    before    │            after             │
                              │      B       │      B        vs base        │
SentryPssAnon/Boot/busybox      7.854Mi ± 0%   7.482Mi ± 0%  -4.73% (n=240)
SentryPssAnon/Boot/python       8.701Mi ± 0%   8.342Mi ± 0%  -4.13% (n=240)
SentryPssAnon/Restore/busybox   9.732Mi ± 0%   9.398Mi ± 1%  -3.43% (n=240)
SentryPssAnon/Restore/python    10.58Mi ± 0%   10.15Mi ± 0%  -4.02% (n=240)
geomean                         9.159Mi        8.785Mi       -4.08%

pkg: boot/restore memory harness, stacked with the lazy seccheck patch (busybox)
                                              │    before    │            after             │
                                              │      B       │      B        vs base        │
SentryPssAnon/Boot/alone                        7.840Mi ± 1%   7.463Mi ± 1%  -4.81% (n=240)
SentryPssAnon/Boot/on-top-of-lazy-seccheck      7.225Mi ± 0%   7.012Mi ± 0%  -2.95% (n=240)
SentryPssAnon/Restore/alone                     9.799Mi ± 0%   9.391Mi ± 0%  -4.17% (n=240)
SentryPssAnon/Restore/on-top-of-lazy-seccheck   9.365Mi ± 0%   9.088Mi ± 0%  -2.96% (n=240)
geomean                                         8.491Mi        8.175Mi       -3.72%

pkg: docker run harness (bridge network, 15-20 idle containers per arm per round)
                                │    before     │                after                 │
                                │       B       │      B        vs base                │
CgroupAnon/busybox                 11.41Mi ± 0%   11.09Mi ± 0%  -2.82% (n=160)
SentryPssAnon/busybox              9.031Mi ± 1%   8.699Mi ± 1%  -3.68% (n=160)
CgroupAnon/python                  11.47Mi ± 1%   11.08Mi ± 1%  -3.39% (n=160)
SentryPssAnon/python               9.084Mi ± 1%   8.740Mi ± 1%  -3.78% (n=160)
CgroupAnon/busybox/cpus=1         10.023Mi ± 1%   9.676Mi ± 1%  -3.47% (p=0.000 n=120)
SentryPssAnon/busybox/cpus=1       8.074Mi ± 1%   7.738Mi ± 1%  -4.16% (p=0.000 n=120)
CgroupAnon/busybox/cpus=0.25       9.936Mi ± 1%   9.570Mi ± 1%  -3.68% (n=120)
SentryPssAnon/busybox/cpus=0.25    7.971Mi ± 1%   7.648Mi ± 1%  -4.04% (n=120)
geomean                            9.543Mi        9.197Mi       -3.63%

                                │   before    │                after                │
                                │   sec/op    │   sec/op     vs base                │
DockerRunTrue/busybox             284.6m ± 1%   283.9m ± 1%       ~ (p=0.836 n=160)
DockerRunTrue/python              284.5m ± 1%   282.5m ± 1%       ~ (p=0.176 n=160)
DockerRunTrue/busybox/cpus=1      284.5m ± 1%   285.4m ± 1%       ~ (p=0.896 n=120)
DockerRunTrue/busybox/cpus=0.25   463.3m ± 2%   462.8m ± 2%       ~ (p=0.938 n=120)
geomean                           321.4m        320.8m       -0.19%

pkg: gvisor.dev/gvisor/test/benchmarks/base
                                                │   before    │                after                │
                                                │   sec/op    │    sec/op     vs base               │
StartupEmpty-16                                   165.9m ± 2%   165.0m ±  1%       ~ (p=0.669 n=16)
StartupNginx-16                                   449.6m ± 1%   451.6m ±  1%       ~ (p=0.590 n=16)
StartupNode-16                                    556.6m ± 1%   554.7m ±  1%       ~ (p=0.239 n=16)
CheckpointEmpty-16                                51.65m ± 2%   51.19m ±  2%       ~ (p=0.809 n=16)
CheckpointNginx-16                                90.45m ± 2%   89.72m ±  2%       ~ (p=0.926 n=16)
RestoreEmpty-16                                   183.7m ± 2%   182.5m ±  2%       ~ (p=0.696 n=16)
RestoreNginx-16                                   230.6m ± 2%   232.0m ±  2%       ~ (p=0.590 n=16)
SizeNginx-16                                      745.6m ± 1%   750.1m ±  1%       ~ (p=0.224 n=16)
SizeNode-16                                       900.4m ± 1%   894.7m ±  1%       ~ (p=0.445 n=16)
Sysbench/testname.CPU-16                          407.6µ ± 3%   407.7µ ±  2%       ~ (p=0.897 n=16)
Sysbench/testname.CPU/threads.16-16               49.58µ ± 3%   50.51µ ±  3%       ~ (p=0.642 n=16)
Sysbench/testname.Memory-16                        1.302 ± 1%    1.292 ±  1%  -0.76% (p=0.006 n=16)
Sysbench/testname.Memory/threads.16-16             1.255 ± 1%    1.256 ±  1%       ~ (p=0.956 n=16)
Sysbench/testname.Mutex-16                        3.687µ ± 1%   3.687µ ±  1%       ~ (p=0.830 n=16)
Syscallbench/syscall.getpid-16                    2.532µ ± 1%   2.559µ ±  2%       ~ (p=0.274 n=16)
Syscallbench/syscall.getpidopt-16                 1.163µ ± 6%   1.195µ ± 11%       ~ (p=0.515 n=16)
Hackbench/ipcMode.pipe/processMode.thread-16      6.738m ± 2%   6.929m ±  3%       ~ (p=0.184 n=16)
Hackbench/ipcMode.socket/processMode.process-16   12.83m ± 7%   12.16m ±  4%       ~ (p=0.119 n=16)
geomean                                           13.31m        13.31m        +0.01%

             │            before            │                        after                        │
             │ average_container_size.bytes │ average_container_size.bytes  vs base               │
SizeNginx-16                   32.27M ± 23%                   31.90M ± 14%       ~ (p=1.000 n=16)
SizeNode-16                    56.34M ±  9%                   54.79M ± 12%       ~ (p=0.926 n=16)
geomean                        42.64M                         41.81M        -1.96%

                                    │            before            │                        after                        │
                                    │ cpu_events.events_per_second │ cpu_events.events_per_second  vs base               │
Sysbench/testname.CPU-16                               3.353k ± 0%                    3.355k ± 0%       ~ (p=0.689 n=16)
Sysbench/testname.CPU/threads.16-16                    26.85k ± 0%                    26.83k ± 1%       ~ (p=0.596 n=16)
geomean                                                9.489k                         9.488k       -0.01%

                                       │              before              │                          after                          │
                                       │ memory_operations.ops_per_second │ memory_operations.ops_per_second  vs base               │
Sysbench/testname.Memory-16                                   1.030M ± 0%                        1.031M ± 0%       ~ (p=0.867 n=16)
Sysbench/testname.Memory/threads.16-16                        1.075M ± 1%                        1.071M ± 1%       ~ (p=0.305 n=16)
geomean                                                       1.052M                             1.051M       -0.12%

                           │          before          │                   after                    │
                           │ average_execution_time.s │ average_execution_time.s  vs base          │
Sysbench/testname.Mutex-16                748.2m ± 2%                762.6m ± 1%  ~ (p=0.072 n=16)

                           │      before       │                after                │
                           │ average_latency.s │ average_latency.s  vs base          │
Sysbench/testname.Mutex-16         748.2m ± 2%         762.6m ± 1%  ~ (p=0.072 n=16)

                           │         before          │                   after                   │
                           │ stddev_execution_time.s │ stddev_execution_time.s  vs base          │
Sysbench/testname.Mutex-16               10.00m ± 0%               10.00m ± 0%  ~ (p=0.484 n=16)

                                                │      before      │                  after                  │
                                                │ execution_time.s │ execution_time.s  vs base               │
Hackbench/ipcMode.pipe/processMode.thread-16           936.0m ± 5%       978.0m ±  4%       ~ (p=0.300 n=16)
Hackbench/ipcMode.socket/processMode.process-16        463.0m ± 7%       481.0m ± 15%       ~ (p=0.485 n=16)
geomean                                                658.3m            685.9m        +4.19%

                                                │          before           │                      after                       │
                                                │ execution_time_per_loop.s │ execution_time_per_loop.s  vs base               │
Hackbench/ipcMode.pipe/processMode.thread-16                    5.785m ± 3%                 5.938m ± 4%       ~ (p=0.210 n=16)
Hackbench/ipcMode.socket/processMode.process-16                 5.662m ± 4%                 5.441m ± 5%       ~ (p=0.180 n=16)
geomean                                                         5.723m                      5.684m       -0.70%

pkg: gvisor.dev/gvisor/test/benchmarks/base (usage_test)
             │   before   │               after               │
             │   sec/op   │   sec/op    vs base               │
SizeNginx-16   2.133 ± 0%   2.133 ± 0%       ~ (p=0.838 n=16)
SizeNode-16    2.106 ± 0%   2.104 ± 0%       ~ (p=0.669 n=16)
SizeEmpty-16   2.092 ± 0%   2.091 ± 0%       ~ (p=0.423 n=16)
geomean        2.110        2.109       -0.04%

             │                before                │                            after                            │
             │ average_container_memory_usage.bytes │ average_container_memory_usage.bytes  vs base               │
SizeNginx-16                            24.12M ± 0%                            23.54M ± 1%  -2.43% (p=0.000 n=16)
SizeNode-16                             49.97M ± 2%                            49.36M ± 1%  -1.23% (p=0.043 n=16)
SizeEmpty-16                            14.91M ± 1%                            14.46M ± 1%  -2.99% (p=0.000 n=16)
geomean                                 26.19M                                 25.61M       -2.22%

pkg: gvisor.dev/gvisor/test/benchmarks/network
                                                               │    before     │                after                │
                                                               │    sec/op     │    sec/op     vs base               │
IperfOneConnection/operation.Upload-16                           1157.6n ± 22%   935.2n ± 50%       ~ (p=0.676 n=16)
IperfOneConnection/operation.Download-16                          863.8n ±  2%   868.5n ±  1%       ~ (p=0.948 n=16)
IperfManyConnections/operation.Upload/parallel.4-16               1.331µ ±  1%   1.331µ ±  2%       ~ (p=0.689 n=16)
IperfManyConnections/operation.Download/parallel.4-16             1.335µ ± 36%   1.335µ ±  1%       ~ (p=0.919 n=16)
IperfManyConnections/operation.Upload/parallel.16-16              1.224µ ±  0%   1.223µ ±  1%       ~ (p=0.860 n=16)
IperfManyConnections/operation.Download/parallel.16-16            1.354µ ±  1%   1.355µ ±  1%       ~ (p=0.963 n=16)
IperfManyConnections/operation.Upload/parallel.64-16              1.251µ ±  1%   1.258µ ±  1%       ~ (p=0.227 n=16)
IperfManyConnections/operation.Download/parallel.64-16            1.439µ ±  1%   1.436µ ±  1%       ~ (p=0.417 n=16)
NginxDocSize/filesize.10Kb/concurrency.1/filesystem.tmpfs-16      209.3µ ±  0%   210.2µ ±  1%       ~ (p=0.160 n=16)
NginxDocSize/filesize.10Kb/concurrency.100/filesystem.tmpfs-16    90.58µ ±  2%   89.60µ ±  2%       ~ (p=0.515 n=16)
NginxDocSize/filesize.1Mb/concurrency.1/filesystem.tmpfs-16       928.6µ ±  0%   933.7µ ±  1%  +0.55% (p=0.039 n=16)
NginxDocSize/filesize.1Mb/concurrency.100/filesystem.tmpfs-16     781.7µ ±  1%   783.6µ ±  1%       ~ (p=0.616 n=16)
geomean                                                           8.039µ         7.904µ        -1.68%

                                                               │      before       │                   after                    │
                                                               │ average_latency.s │ average_latency.s  vs base                 │
NginxDocSize/filesize.10Kb/concurrency.1/filesystem.tmpfs-16           200.0µ ± 0%         200.0µ ± 0%       ~ (p=1.000 n=16) ¹
NginxDocSize/filesize.10Kb/concurrency.100/filesystem.tmpfs-16         3.350m ± 4%         3.300m ± 3%       ~ (p=0.695 n=16)
NginxDocSize/filesize.1Mb/concurrency.1/filesystem.tmpfs-16            900.0µ ± 0%         900.0µ ± 0%       ~ (p=1.000 n=16) ¹
NginxDocSize/filesize.1Mb/concurrency.100/filesystem.tmpfs-16          64.30m ± 2%         64.60m ± 2%       ~ (p=0.462 n=16)
geomean                                                                2.495m              2.489m       -0.26%
¹ all samples are equal

                                                       │    before     │                after                 │
                                                       │      B/s      │      B/s       vs base               │
IperfOneConnection/operation.Upload-16                   109.6Gi ± 22%   130.6Gi ± 34%       ~ (p=0.669 n=16)
IperfOneConnection/operation.Download-16                 141.3Gi ± 37%   140.6Gi ± 10%       ~ (p=0.956 n=16)
IperfManyConnections/operation.Upload/parallel.4-16      91.72Gi ±  1%   91.71Gi ±  1%       ~ (p=0.696 n=16)
IperfManyConnections/operation.Download/parallel.4-16    91.50Gi ±  1%   91.45Gi ±  1%       ~ (p=0.897 n=16)
IperfManyConnections/operation.Upload/parallel.16-16     99.78Gi ±  1%   99.84Gi ±  1%       ~ (p=0.838 n=16)
IperfManyConnections/operation.Download/parallel.16-16   90.13Gi ±  1%   90.06Gi ±  0%       ~ (p=1.000 n=16)
IperfManyConnections/operation.Upload/parallel.64-16     97.59Gi ±  1%   97.06Gi ±  1%       ~ (p=0.224 n=16)
IperfManyConnections/operation.Download/parallel.64-16   84.87Gi ± 25%   85.00Gi ± 25%       ~ (p=0.445 n=16)
geomean                                                  99.59Gi         101.7Gi        +2.08%

                                                       │           before           │                       after                       │
                                                       │ bandwidth.bytes_per_second │ bandwidth.bytes_per_second  vs base               │
IperfOneConnection/operation.Upload-16                                 3.801G ±  7%                 3.638G ±  6%       ~ (p=0.445 n=16)
IperfOneConnection/operation.Download-16                               4.378G ± 68%                 4.533G ± 32%       ~ (p=0.239 n=16)
IperfManyConnections/operation.Upload/parallel.4-16                    1.446G ±  5%                 1.446G ±  4%       ~ (p=0.401 n=16)
IperfManyConnections/operation.Download/parallel.4-16                  1.440G ±  5%                 1.507G ±  9%       ~ (p=0.564 n=16)
IperfManyConnections/operation.Upload/parallel.16-16                   790.3M ±  0%                 788.6M ±  0%       ~ (p=0.210 n=16)
IperfManyConnections/operation.Download/parallel.16-16                 564.3M ±  8%                 563.0M ± 17%       ~ (p=0.616 n=16)
IperfManyConnections/operation.Upload/parallel.64-16                   199.9M ±  7%                 202.7M ± 12%       ~ (p=0.752 n=16)
IperfManyConnections/operation.Download/parallel.64-16                 135.0M ± 15%                 149.4M ± 18%       ~ (p=0.128 n=16)
geomean                                                                896.4M                       913.0M        +1.86%

                                                               │         before          │                     after                      │
                                                               │ requests_per_second.QPS │ requests_per_second.QPS  vs base               │
NginxDocSize/filesize.10Kb/concurrency.1/filesystem.tmpfs-16                 6.433k ± 0%               6.419k ± 0%       ~ (p=0.078 n=16)
NginxDocSize/filesize.10Kb/concurrency.100/filesystem.tmpfs-16               26.77k ± 4%               26.48k ± 3%       ~ (p=0.897 n=16)
NginxDocSize/filesize.1Mb/concurrency.1/filesystem.tmpfs-16                  1.143k ± 0%               1.141k ± 0%       ~ (p=0.273 n=16)
NginxDocSize/filesize.1Mb/concurrency.100/filesystem.tmpfs-16                1.375k ± 2%               1.375k ± 3%       ~ (p=0.831 n=16)
geomean                                                                      4.056k                    4.040k       -0.39%

Benchmark environment

GCE c4-standard-16, COS cos-129-19506-448-8 (kernel 6.12.105+), hyperdisk-balanced, docker 27.4.1, cgroup v2. before is master at 2511bc60218a, after is the patch above. Both are opt release builds. The boot/restore memory harness runs busybox (sleep) or python (http.server) sandboxes directly with runsc run, or restores them from one checkpoint taken with master. It reads the Sentry's Pss_Anon from smaps after 20 s, over 8 rounds in ABBA order with caches dropped each round. The stacked block uses the same harness with four arms: master vs this patch (alone), and the lazy seccheck patch vs both patches (on-top-of-lazy-seccheck). The docker harness times docker run --rm <image> true. It then starts 15-20 idle containers per arm per round (8 rounds, 0.3-1.5 s apart, with --cpus where shown) and reads cgroup anon and Sentry Pss_Anon after 20 s. Memory-harness samples are per sandbox, so they are grouped by round rather than fully independent. The //test/benchmarks blocks are the real targets, run as docker runtimes in ABBA order with one invocation per sample. usage_test is shown separately because it reuses size_test's benchmark names. size_test's SizeEmpty fails on both arms with an upstream harness error, so it is omitted.

Activity

No activity yet.