#819

TSI: accept() holds the socket lock across the blocking vsock round-trip — concurrent getsockname() on the listening socket never returns

Openjmagderopened 2mo ago

Environment

  • libkrun 1.19.4 (Arch package libkrun 1.19.4-1), libkrunfw 5.5.0-1, crun-based krun 1.29.1-1
  • Host: Arch Linux x86_64, kernel 6.12.91, KVM; rootless podman 5.x with pasta
  • Guest: python:3.12-slim (bookworm, glibc 2.36)
  • Networking: default — no NIC, so implicit TSI with TsiFlags::HIJACK_INET (tsi_hijack on the guest kernel command line)
  • The guest-side code involved is libkrunfw's kernel patch net/tsi/af_tsi.c, which is byte-identical between the v5.5.0 tag and current libkrunfw main — filing here (not libkrunfw) following the precedent of #511.

Summary

The guest TSI accept() path holds lock_sock(sk) on the listening socket while blocking on the vsock round-trip to the VMM. Any concurrent getsockname() on the same listening socket therefore blocks forever. On non-TSI Linux, accept() does not hold the socket lock across blocking, which is why the spawn-accept-thread / getsockname pattern (standard in debugpy's serve(), and in many servers) works everywhere else.

This reproduces on a fixed-port listener with no inbound connection at all — merely having one thread blocked in accept() while another thread calls getsockname() on the same socket deadlocks. It is independent of the ephemeral-port divergence reported in #818 .

Minimal repro (tested: host finishes instantly, krun guest hangs)

cat > d.py <<'EOF'
import socket, threading, time
s = socket.socket(); s.bind(('0.0.0.0', 45678)); s.listen(5)
t = threading.Thread(target=s.accept, daemon=True)  # blocks in guest TSI accept
t.start()
time.sleep(0.5)
print('calling getsockname...', flush=True)
print('getsockname:', s.getsockname(), flush=True)  # never returns under TSI
print('SCRIPT DONE', flush=True)
EOF

python3 d.py    # host sanity: prints everything and exits

podman run --rm --runtime=krun \
    --annotation krun.ram_mib=1024 --annotation krun.cpus=2 \
    --mount type=bind,src=$PWD/d.py,target=/d.py \
    python:3.12-slim timeout 15 python3 /d.py

Host output:

calling getsockname...
getsockname: ('0.0.0.0', 45678)
SCRIPT DONE

krun guest output (killed by timeout 15; would hang forever):

calling getsockname...

Real-world capture

Seen while debugging why debugpy hangs inside a krun container (daemonized adapter process, strace -f):

81  ... bind(4, {AF_INET, 0.0.0.0:5678}, 16) = 0
81  ... listen(4, 4096) = 0
81  ... clone3(...) = 82          <- debugpy serve() spawns its accept thread
82  ... (accept path, blocks on the vsock round-trip to the VMM)
81  ... getsockname(4,  <unfinished ...>   <- never returns

Relation to #511

Same defect class as #511 ("severe lock contention when creating/destroying connections" — 3 conn/s with 90% loss under TSI vs 2747–5747 conn/s with pasta/passt). This report shows the contention is not only a performance issue: it can deadlock unrelated syscalls operating on the same socket.

Suggested fix direction

Don't hold sk_lock across the blocking VMM wait — e.g. release/reacquire around the round-trip, or use a separate lock for proxy state.

Activity

2
jmagder2mo ago

Follow-up: same investigation, same result with a real NIC — with --annotation krun.use_passt=1 the d.py repro prints getsockname and exits normally, so the deadlock is TSI-specific. Consistent with the pasta/passt contrast in #511's benchmarks.

djbclark16d ago

Same lock, wider reach (measured on SIPB's Hyades runner, libkrun "2.0-sipb", guest kernel 6.12.91; details in #926):

  1. A blocking recv()/read() holds it as well as accept(), and the calls it blocks include send(), shutdown() and getpeername(), not only getsockname(). duplex-probe.c: a thread parked in recv() on a connected socket and another thread send()ing on it: the data reaches the peer only after the recv() is released (normal kernel: 0 to 1 ms).
  2. A thread parked in poll() does not hold the lock, so from userspace one can wait in poll() before a blocking read/accept; an LD_PRELOAD shim doing that made a 1,417-test Gradle/okhttp suite on that runner go from 7 failures per run to 1 (the remaining one also fails elsewhere).
  3. A listener closed while another thread is blocked in accept() is not released; a new listener on the same port then loses most connections: #929.