TSI: a blocking recv()/read()/accept() blocks send(), shutdown() and getpeername() on the same socket from other threads (hangs Java ServerSocket.close(), stalls WebSocket clients)

Open

djbclarkopened 16d ago

Summary

In a krun guest with TSI networking, shutdown(2) on a TCP socket never returns while another thread of the process is blocked in accept() (listener) or read() (connected client) on that same socket. getpeername() on the listener blocks the same way. (getsockopt(SO_ACCEPTCONN) does not block, but returns 0 for a socket that is listening.) On a normal Linux kernel shutdown() on a listener wakes the blocked accept(); it is how Java (ServerSocket.close(), Socket.close()), okhttp's MockWebServer and many other runtimes unblock an accept thread. Here the closing thread hangs forever, so for example a Gradle test JVM stops with a thread stuck in sun.nio.ch.Net.shutdown and the CI job runs until its timeout. Related to the port-0 problem in #818, found while working around it.

Prepared with an AI coding assistant (Claude Code) from probe runs on SIPB's Hyades CI runner (libkrun "2.0-sipb", guest kernel 6.12.91, root and loopback through TSI); I reviewed the results. I have not tried upstream main outside that runner.

Repro (inside a krun container)

import socket, threading, time
s = socket.socket(); s.bind(("127.0.0.1", 31337)); s.listen(5)
threading.Thread(target=lambda: s.accept(), daemon=True).start(); time.sleep(0.3)
s.shutdown(socket.SHUT_RDWR)   # never returns

Also: ServerSocket s = new ServerSocket(31340) with a thread in s.accept(), then s.close() from another thread: hangs. A client Socket with a thread blocked in getInputStream().read(), then close(): hangs.

What I measured (C probe `accept-probe.c`, thread blocked in `accept()` on a loopback listener)

action from the other thread	result
`pthread_kill(SIGUSR1)`	`accept` returns `EINTR` (works)
`dup2(/dev/null, fd)` then `pthread_kill`	`accept` returns `EINTR` (works; this is what the JDK does after `shutdown`)
`close(fd)` alone	`accept` stays blocked (same as a normal kernel, where close does not wake accept)
`shutdown(fd, SHUT_RD)`	never returns
`shutdown(fd, SHUT_RDWR)`	never returns
`getpeername(fd)`	never returns
`getsockopt(fd, SOL_SOCKET, SO_ACCEPTCONN)`	returns at once, value 0 (the socket is listening, so this looks wrong)
`shutdown(fd, SHUT_RDWR)` on a listener with no thread in `accept()`	fails with `ENOTCONN` (a normal kernel returns 0)

So the blocked accept() seems to hold a lock that some other socket operations need. Probe workflows and logs: https://forgejo.mit.edu/djbclark/loopback-port-shim (ci/probes/probe-shutdown*.yml, ci/probes/probe-accept.yml, ci/tools/accept-probe.c).

Workaround in use

An LD_PRELOAD shim records listening fds in listen() (forgotten in close/dup2/dup3), makes shutdown() a no-op on them, and runs shutdown() on other sockets in a helper thread with a 300 ms wait. The JDK's dup2 plus signal then wakes the blocked reader. With it a 1,417-test Gradle/Robolectric suite completes. Roughly six MockWebServer tests still fail with unexpected end of stream (a different subset each run), also with shutdown() made a no-op, so there may be a second TSI loopback issue; I do not have a minimal repro for it yet.

Questions

Is the blocking intended (a socket-wide lock held during accept/read), or a bug? A fix that lets shutdown and getpeername proceed concurrently with a blocked accept would match Linux behavior.
For CI users, would krun.use_passt=1 avoid TSI for loopback entirely? SIPB prefers not to enable it (nested passt), which is why I am asking about a TSI fix.

#818: listeners bound to port 0 are unreachable under TSI.
#924: virtiofs performance on the same runner.

Search for existing issues

Searched libkrun/libkrun issues (open and closed) for "shutdown accept", "TSI hang", "getpeername" and "tsi blocking accept lock" before filing: nothing nearer than #818 (port-0 listeners) and #924 (virtiofs speed).

Correction (same day): an earlier version of this issue also said getsockopt() blocks; it does not, it returns at once but reports SO_ACCEPTCONN as 0 for a listening socket (probe run 206).

Activity

djbclark16d ago

Update, 2026-10-14 (later). First a correction: after filing I found #819, which reports the same root cause (the guest TSI accept() holds lock_sock across the blocking vsock round trip, so getsockname() blocks). This issue is a near-duplicate; I keep it open only as the wider list of calls the lock affects, and I am fine with it being closed in favour of #819. (My earlier search used "shutdown accept" and similar and missed it.)

New measurements (runner libkrun "2.0-sipb", guest kernel 6.12.91; control: the same programs on Ubuntu 24.04, Linux 6.18.15):

A blocking recv()/read() holds the lock too, and what it blocks includes send(). duplex-probe.c: one thread is parked in recv() on a connected socket (tested on the client end and on the accepted end), a second thread send()s on the same socket. Nothing reaches the peer until the parked recv() is released (the probe releases it after 3 s; send() then returns after 3004 ms). Normal kernel: 0 to 1 ms. Any client with a reader thread and a writer thread stalls: okhttp's WebSocket client was the case that found it (three tests in a 1,417-test suite timed out deterministically).
A thread parked in poll() does not hold the lock. With the reader parked in poll(-1), in poll with a 20 ms timeout, or in a non-blocking recv loop, send() reaches the peer in 0 to 1 ms (same probe, reader=poll|pollslice|nonblock). So waiting in poll() before a blocking read/accept avoids all of it from userspace; the LD_PRELOAD shim loopback-port-shim.c now does exactly that (read, recv, recvfrom, recvmsg, accept, accept4; signals still interrupt the poll, so the JDK's close wake-up still works).
Result. The full Plektra Gradle/Robolectric suite (1,417 tests) on the runner: 7 failures per run (one known unrelated test plus about six flaky MockWebServer/WebSocket tests, a different subset each time) before the shim changes; 1 failure (a test that also fails on GitHub Actions and on a laptop) in two consecutive runs after (runs 229 and 230). A standalone okhttp 4.11 MockWebServer + WebSocket echo loop went from 0 of 60 echoes to 60 of 60, and 800 HTTP round trips from 150 s to 1 s.
The "second TSI loopback issue" I mentioned in the description was partly my own shim (it gave consecutive listeners the same random port within one second) and partly a separate libkrun behaviour, filed as #929.

Question: could the guest TSI driver wait for the vsock reply without holding the socket lock, the way tcp_recvmsg waits with sk_wait_data (which drops the lock while sleeping)? I have not tried a kernel patch.

djbclark5d ago

Another symptom of this class, seen on SIPB Hyades (libkrun TSI): tailscaled --tun=userspace-networking (Tailscale 1.104.2) hangs after logging "link state"; the guest has only dummy0. One thread sits in the kernel's __lock_sock and the container needed a force-kill. It looks like the same cause as this issue (a blocking call holds the socket lock while other operations on that socket wait), but I have not confirmed that. Happy to test a fix.