#818

TSI: bind(port 0) + listen() lands the VMM proxy on a different ephemeral port than the guest — ephemeral listeners unreachable

Openjmagderopened 2mo ago

Environment

  • libkrun 1.19.4 (Arch package libkrun 1.19.4-1), libkrunfw 5.5.0-1, crun-based krun 1.29.1-1
  • Host: Arch Linux x86_64, kernel 6.12.91, KVM; rootless podman 5.x with pasta
  • Guest: python:3.12-slim (bookworm, glibc 2.36)
  • Networking: default — no NIC, so implicit TSI with TsiFlags::HIJACK_INET (tsi_hijack on the guest kernel command line)
  • Verified still present in current main (2.0 dev): TsiListenRsp { result: i32 } in src/devices/src/virtio/vsock/packet.rs is identical there — no address is communicated back in either version.

Summary

bind(port 0) followed by listen() under TSI leaves the listener unreachable at its own port, including for the process that created it:

  1. Guest app calls bind(127.0.0.1:0); the guest kernel assigns port X.
  2. listen() sends TSI_LISTEN to the VMM with the recorded request address (port 0) — net/tsi/af_tsi.c:tsi_listen() sends tsk->bound_addr, which still contains port 0.
  3. The VMM (src/devices/src/virtio/vsock/tsi_stream.rs:try_listen()) binds that address in the VMM process's own netns, where bind(0) gets a different port Y from that netns's ephemeral allocator.
  4. TsiListenRsp returns only a result code — Y is never communicated back, so the guest keeps reporting X in getsockname().

Result: connect(127.0.0.1:X) gets ECONNREFUSED. The listener is alive at port Y, unreachable and unknown.

Minimal repro (bare krun container; tested 3/3)

cat > t.py <<'EOF'
import socket, time
s = socket.socket(); s.bind(('127.0.0.1', 0))
port = s.getsockname()[1]; s.listen(1)
print('bound:', port)
time.sleep(0.5)
c = socket.socket(); c.settimeout(5)
c.connect(('127.0.0.1', port))   # -> ConnectionRefusedError
EOF

podman run --rm --runtime=krun \
    --annotation krun.ram_mib=1024 --annotation krun.cpus=2 \
    --mount type=bind,src=$PWD/t.py,target=/t.py \
    python:3.12-slim python3 /t.py

Output:

bound: 43859
ConnectionRefusedError: [Errno 111] Connection refused

Control: the same script with a fixed port (bind(('127.0.0.1', 45677)), connect to 45677) succeeds — fixed-port binds do not diverge, which confirms the mechanism.

Direct proof of the divergence

Run the same ephemeral listener inside a --network host krun container and inspect the host side while the guest sleeps on accept():

guest: bound 127.0.0.1:53603 (X)
host:  ss -ltn  →  LISTEN 127.0.0.1:45551 (Y — assigned by the VMM's netns)
connect to X → refused; connect to Y → accepted, guest accept() fires

Real-world impact

debugpy (python -m debugpy --listen 0.0.0.0:5678 …) always creates a 127.0.0.1:0 endpoints listener whose adapter child connects back, so every debugpy session dies after 30 s with RuntimeError: timed out waiting for adapter to connect. Reproduced in a bare krun container with debugpy alone. Anything else using ephemeral listeners is similarly affected.

Suggested fix directions

Return the actually-bound address/port in TsiListenRsp and have the guest update the socket's local address; or have the guest request a concrete port for the shadow bind (it already knows X). We currently carry a userspace workaround (LD_PRELOAD rewriting port-0 binds to concrete ports) and are happy to contribute a VMM/kernel fix if this direction sounds right.

Related: #819 — same investigation, TSI locking bug

Activity

3
jmagder2mo ago

Follow-up: for anyone hitting this, booting with --annotation krun.use_passt=1 (real virtio-net NIC, TSI fully disabled) makes the symptom disappear — the same t.py succeeds with no other changes, which also confirms the bug is TSI-specific. We run our nested-podman workload on passt now.

jmagder2mo ago

I think I have a working patch to fix this issue for the included 6.12.91 kernel. I'm going to revisit it on a fresh mind tomorrow, and if I still think it makes sense I will make a PR against libkrunfw.

BroadlyWhitaker15d ago

I think this explains failures I've been seeing with omp coding agent running inside a podman container with krun as runtime. I see ECONNREFUSED errors when a generated script tries to connect to the PI_TOOL_BRIDGE_* listener it spawns.

Switching podman to passt does cure it, but it breaks the default route init when booting the container, if wireguard is in use. podman doesn't currently let one pass -i <interface> and -g <gateway> to override the auto config, so, I just pass in the gw ip via env var and the login script does the ip config.