GKE Autopilot 1.37.1 (COS 6.18.48, containerd 2.4.1): node freezes after the efficiency-daemon widens the kubepods cpuset under a busy systrap sandbox

Issues/google/gvisor/#16475

GKE Autopilot 1.37.1 (COS 6.18.48, containerd 2.4.1): node freezes after the efficiency-daemon widens the kubepods cpuset under a busy systrap sandbox

Open

ReadAccessBotopened 3d ago

Description

On GKE Autopilot nodes at 1.37.1-gke.1697000 (Container-Optimized OS, kernel 6.18.48+, containerd 2.4.1) a node running one busy gVisor sandbox dies seconds to about 140 s after the Autopilot efficiency-daemon widens the cpuset of kubepods.slice from 2 to 4 enabled cores. The daemon (an NRI plugin, v36.3.0-gke.13, drivers cpu_limits, memory_limits, cpuset_cpus, ek_cpu_limits) recomputes cpu.max, memory.max and the enabled cores of kubepods.slice from the pods' requests at every RunPodSandbox and StopPodSandbox, so starting a second pod of ordinary size beside the sandbox triggers it. The kube-node-lease stops renewing, Ready goes Unknown about 40 s later, and the node hangs until GKE auto-repair replaces it.

Expected: a live cpuset change of the pods' cgroup is transparent to a running sandbox. Observed: the node dies.

A = a sandbox burning one core and rewriting 2.2 GB with fsync; B = a pod started beside it 30 s later, on a node 1 to 1.5 min old. On the 1.37.1 image, A busy:

what the daemon changed at B's start	B	crashed
cpuset 2 to 4, cpu.max +1000m, memory.max +4 GiB (B 1 CPU / 4 GiB)	runsc	3 of 3: two within 5 s, one after 2 min 20 s
the same (B 1 CPU / 4 GiB)	plain runc	1 of 1, after 2 min 23 s
cpuset 2 to 4, cpu.max +616m, memory.max +4 GiB (B 50m / 4 GiB)	runsc	1 of 1, within 3 s
cpuset 2 to 4, cpu.max +1000m, memory.max +1 GiB (B 1 CPU / 64 MiB)	runsc	1 of 1, within 5 s
cpuset unchanged: cpu.max +50m, memory.max +60 MiB (B 50m / 64 MiB)	runsc	0 of 2
cpuset unchanged (A requested 1500m, so 4 cores from A's own start): cpu.max +1000m, memory.max +4 GiB	runsc	0 of 2
cpuset 2 to 4 and the rest, but A idle	runsc	0 of 1

The same widening under the same load on the 1.36.4-gke.2377000 image (kernel 6.12.110+, containerd 2.2.7): 0 of 4. The same load on a 1.37.1 node without the daemon (a node-billed e2-standard-8, which the daemon's DaemonSet does not schedule on): 0 of 1. A alone with this load or a lighter one, no second pod: 0 of 7 in 4.5 min each.

So neither a second sandbox nor the raise of cpu.max or memory.max triggers it; the cpuset change under a busy sandbox on this image does. Not tested: A as a plain runc container, so whether the sandbox is required is open. One node died 17 s after A's own start, before any second pod, after system pods came and went on the fresh node; it is the only one that rebooted (new bootID) rather than hung, and we cannot place it.

The daemon's last lines on an e2-standard-4 that crashed (timestamp and message only):

2026-10-26T18:50:24.202920852Z nri: Start - RunPodSandbox, pod [namespace=<ns>,name=<pod-a>]
2026-10-26T18:50:24.207601951Z CPUSet mutation: enabled [0 2], disabled [1 3]
2026-10-26T18:50:24.207935921Z Actual limits: {Millicores: 1547, Memory: 5.44 Gi} (enabled cores via cpuset: 2) | Enabled drivers: [cpu_limits memory_limits cpuset_cpus ek_cpu_limits] | Host: {Millicores: 4000, Memory: 15.62 Gi} | Reserved: {Millicores: 80, Memory: 2.76 Gi}
2026-10-26T18:51:21.499811835Z nri: Start - RunPodSandbox, pod [namespace=<ns>,name=<pod-b>]
2026-10-26T18:51:21.508778326Z Inflating driver: cpuset_cpus
2026-10-26T18:51:21.508856106Z CPUSet mutation: enabled [0 2 1 3], disabled []
2026-10-26T18:51:21.509919396Z Actual limits: {Millicores: 2163, Memory: 9.44 Gi} (enabled cores via cpuset: 4) | Enabled drivers: [cpu_limits memory_limits cpuset_cpus ek_cpu_limits] | Host: {Millicores: 4000, Memory: 15.62 Gi} | Reserved: {Millicores: 80, Memory: 2.76 Gi}
2026-10-26T18:51:21.510080306Z nri: Done - RunPodSandbox, pod [namespace=<ns>,name=<pod-b>]

B's container started 18:51:24; the node's last lease renewal was 18:51:22.55; NodeNotReady 18:52:16.

Steps to reproduce

An Autopilot cluster at 1.37.1-gke.1697000 (Rapid channel; ours was upgraded to it). GKE Sandbox needs no cluster flag on Autopilot, only runtimeClassName: gvisor in the pod. Do not set a cloud.google.com/machine-family selector: it puts the pod on a node-billed node without the daemon. Without one, Autopilot gave us ek-standard-8 (4 of 4 crashed) or e2-standard-4 (2 of 2).

kubectl create namespace freeze-repro; apply A below. With no other gVisor pod in the cluster it gets a new node from node auto-provisioning (1 to 2 min). Wait until Running.
30 s later apply B: required podAffinity to A, 1 CPU / 4 GiB. On ek-standard-8 a GKE system pod (gke-system-balloon-pod) takes the node's spare room about 10 min after its last pod change and B stays Pending after that, so apply B inside that window.
Watch from before B: the node's Ready condition; the lease (kubectl -n kube-node-lease get lease <node> -o jsonpath='{.spec.renewTime}', renewed every 10 s); and the daemon's log on that node (kubectl -n kube-system logs -f <efficiency-daemon pod on the node> -c efficiency-daemon), followed live, because a frozen node's last roughly 20 s of logs are never shipped.
The lease stops within seconds or after about 140 s of the daemon's enabled cores via cpuset: 4 line; Ready goes Unknown about 40 s later; B is stuck in ContainerCreating or runs until the node dies; auto-repair recycles the node after about 10 min. A control that survives 5 min has survived.

Controls that did not crash: B at cpu: 50m, memory: 64Mi (cpuset stays at 2 cores); A at cpu: 1500m (4 cores from A's own start); A with sleep infinity. B without runtimeClassName (plain runc) does crash.

We ran another image with the same tools (sh, python3, dd); python:3.12-slim is untested but has them. The PVC, the emptyDir, fsGroup and the container securityContext are what we ran; the load touches only /tmp and we have not tried without them.

Manifest A:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pod-a-work
  namespace: freeze-repro
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: standard-rwo
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-a
  namespace: freeze-repro
  labels: {app: freeze-repro, name: pod-a, role: a}
spec:
  runtimeClassName: gvisor
  automountServiceAccountToken: false
  restartPolicy: Never
  nodeSelector:
    sandbox.gke.io/runtime: gvisor
  tolerations:
    - key: sandbox.gke.io/runtime
      operator: Equal
      value: gvisor
      effect: NoSchedule
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels: {app: freeze-repro, role: a}
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch
  containers:
    - name: main
      image: python:3.12-slim
      command: ["/bin/sh", "-c"]
      args:
        - |
          python3 -c 'while True: pass' & while :; do for i in 1 2 3 4 5 6 7 8 9 10 11; do dd if=/dev/zero of=/tmp/f$i bs=1M count=200 conv=fsync 2>/dev/null; done; done
      securityContext:
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        runAsUser: 1000
        capabilities:
          drop: ["ALL"]
      resources:
        requests: {cpu: "1000m", memory: "4Gi", ephemeral-storage: "10Gi"}
        limits: {cpu: "1000m", memory: "4Gi", ephemeral-storage: "10Gi"}
      volumeMounts:
        - {name: work, mountPath: /work}
        - {name: scratch, mountPath: /run/scratch}
  volumes:
    - name: scratch
      emptyDir:
        medium: Memory
        sizeLimit: 1Mi
    - name: work
      persistentVolumeClaim:
        claimName: pod-a-work

Manifest B, the same shape, idle, with podAffinity to A in place of the anti-affinity:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pod-b-work
  namespace: freeze-repro
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: standard-rwo
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-b
  namespace: freeze-repro
  labels: {app: freeze-repro, name: pod-b, role: b}
spec:
  runtimeClassName: gvisor
  automountServiceAccountToken: false
  restartPolicy: Never
  nodeSelector:
    sandbox.gke.io/runtime: gvisor
  tolerations:
    - key: sandbox.gke.io/runtime
      operator: Equal
      value: gvisor
      effect: NoSchedule
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels: {app: freeze-repro, name: pod-a}
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch
  containers:
    - name: main
      image: python:3.12-slim
      command: ["sleep", "infinity"]
      securityContext:
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        runAsUser: 1000
        capabilities:
          drop: ["ALL"]
      resources:
        requests: {cpu: "1000m", memory: "4Gi", ephemeral-storage: "10Gi"}
        limits: {cpu: "1000m", memory: "4Gi", ephemeral-storage: "10Gi"}
      volumeMounts:
        - {name: work, mountPath: /work}
        - {name: scratch, mountPath: /run/scratch}
  volumes:
    - name: scratch
      emptyDir:
        medium: Memory
        sizeLimit: 1Mi
    - name: work
      persistentVolumeClaim:
        claimName: pod-b-work

runsc version

runsc -version cannot be run on an Autopilot node. The banner the runsc-metric-server sidecar in kube-system logs (the same /sbin/runsc binary), node image 1.37.1-gke.1697000:

Version google-983538519, go1.28-20260721-RC01 cl/951519500 +3ebc191975 X:fieldtrack,boringcrypto,simd,mapsplitgroup, amd64, 2 CPUs, linux
Platform: systrap
FileAccess: exclusive / Directfs: true / Overlay: root:self
Network: sandbox

docker version (if using docker)

Not used.

uname

No node shell on Autopilot. Node status: Linux 6.18.48+, amd64, Container-Optimized OS from Google.

kubectl (if using Kubernetes)

kubeletVersion:          v1.37.1-gke.1697000
kernelVersion:           6.18.48+
osImage:                 Container-Optimized OS from Google
containerRuntimeVersion: containerd://2.4.1
instance-type:           ek-standard-8, e2-standard-4 (both crash)
sandbox.gke.io/runtime:  gvisor

control image that never froze:
kubeletVersion v1.36.4-gke.2377000, kernelVersion 6.12.110+, containerd://2.2.7

repo state (if built from source)

Not built from source; GKE-managed runsc, see above.

runsc debug logs (if available)

None. Autopilot gives no node shell, no serial console and no nodes/proxy, and kubectl debug node --profile=sysadmin is refused. A hung node ships nothing from its last roughly 20 s and auto-repair recreates it with a new disk. On one node of this image whose serial console we could read, the kernel runs with nmi_watchdog=0 and the e2 CPU has no PMU, so a hard lockup there leaves no trace by construction.

Question

Is this runsc (systrap) reacting to a live cpuset change of its cgroup, or the kernel or containerd of this node image? What we can show: a busy systrap sandbox, the kubepods.slice cpuset widened underneath it, and the 1.37.1 image; remove the widening, the load or the image and no round crashed. If it is the node image rather than runsc, say so and we take it to the GKE tracker.

Ask: a fix, or that GKE Sandbox nodes be exempt from live cpuset widening by the efficiency-daemon. We will run the reproduction again with any instrumentation you give us.

Steps to reproduce

No response

runsc version

undefined

docker version (if using docker)

undefined

uname

No response

kubectl (if using Kubernetes)

undefined

repo state (if built from source)

No response

runsc debug logs (if available)

undefined

Activity

No activity yet.