#622

UDPProxy accumulates open sockets indefinitely under high-frequency UDP load (DNS, NTP) due to 90s connection tracking timeout

OpenChad-Ballayopened 7mo ago

Description

I want to be upfront: I'm not a Go developer and I'm using an AI coding assistant to help write this report. The data and observations are my own.

When containers generate high-concurrency UDP traffic where each packet uses a new ephemeral source port (DNS queries, NTP), UDPProxy accumulates open sockets indefinitely. Each unique source port is a cache miss in the connection tracking table, creating a new net.Conn and a new replyLoop goroutine held open for the full UDPConnTrackTimeout of 90 seconds. At any meaningful concurrency this compounds into thousands of open sockets and eventual CPU explosion on the hostagent process.

A PR with a minimal fix has been submitted: https://github.com/containers/gvisor-tap-vsock/pull/621


Environment

FieldValue
gvisor-tap-vsock versionv0.8.6 (bundled in Rancher Desktop 1.22.0 / Lima 1.2.1)
macOS26.3 (Tahoe)
VM typevz (Apple Virtualization Framework) + vzNAT
HardwareApple M4 Pro

Steps to Reproduce

  1. Run a container making 50+ concurrent async DNS A-record lookups against an external nameserver
  2. Monitor the limactl hostagent process:
# Get PID
pgrep -f "limactl hostagent"

# Watch FD count climb
watch -n2 "lsof -p <PID> | wc -l"

Actual Behavior

File descriptors accumulate without bound. Each DNS query opens a new UDP socket that is held open for 90 seconds. At MAX_CONCURRENT=50, the FD count grows from a baseline of ~23 to 5,000+ within minutes.

Total open FDs:       5,187
Total UDP sockets:    5,282  (all to DNS servers)
limactl CPU:          1,171%

Root Cause

pkg/services/forwarder/udp_proxy.go:

const (
    UDPConnTrackTimeout = 90 * time.Second
)

DNS resolvers use a new ephemeral source port per query → always a cache miss in connTrackTable → new net.Conn + new replyLoop goroutine per query, each blocking for 90s.

This is not DNS-specific. NTP traffic exhibits identical behavior (independently reported by another user on https://github.com/lima-vm/lima/issues/4666).


Confirmation Test

I ran a controlled test pausing 60s, 90s, 120s, and 150s after each burst of 500 DNS queries:

PauseFDs after pause
60s~520 — no recovery
90s23 — full baseline restored
120s23 — full baseline restored
150s23 — full baseline restored

The drain begins at precisely ~87-91 seconds after each burst ends, every time, confirming UDPConnTrackTimeout = 90s as the sole mechanism.


Fix

See PR #621. Reduces UDPConnTrackTimeout to 10 seconds. DNS and NTP responses are immediate — there is no reason to hold connections open for 90 seconds for these protocols. A more sophisticated solution (configurable timeout, per-protocol handling, or a bounded connection table with eviction) may be preferable long-term.


Related

Activity

0

No activity yet.