Environment
- libkrun 1.19.4 (Arch package
libkrun 1.19.4-1), libkrunfw 5.5.0-1, crun-based krun 1.29.1-1 - Host: Arch Linux x86_64, kernel 6.12.91, KVM; rootless podman 5.x with pasta
- Guest:
python:3.12-slim(bookworm, glibc 2.36) - Networking: default — no NIC, so implicit TSI with
TsiFlags::HIJACK_INET(tsi_hijackon the guest kernel command line) - Verified still present in current
main(2.0 dev):TsiListenRsp { result: i32 }insrc/devices/src/virtio/vsock/packet.rsis identical there — no address is communicated back in either version.
Summary
bind(port 0) followed by listen() under TSI leaves the listener
unreachable at its own port, including for the process that created it:
- Guest app calls
bind(127.0.0.1:0); the guest kernel assigns port X. listen()sendsTSI_LISTENto the VMM with the recorded request address (port 0) —net/tsi/af_tsi.c:tsi_listen()sendstsk->bound_addr, which still contains port 0.- The VMM (
src/devices/src/virtio/vsock/tsi_stream.rs:try_listen()) binds that address in the VMM process's own netns, wherebind(0)gets a different port Y from that netns's ephemeral allocator. TsiListenRspreturns only a result code — Y is never communicated back, so the guest keeps reporting X ingetsockname().
Result: connect(127.0.0.1:X) gets ECONNREFUSED. The listener is alive at
port Y, unreachable and unknown.
Minimal repro (bare krun container; tested 3/3)
cat > t.py <<'EOF'
import socket, time
s = socket.socket(); s.bind(('127.0.0.1', 0))
port = s.getsockname()[1]; s.listen(1)
print('bound:', port)
time.sleep(0.5)
c = socket.socket(); c.settimeout(5)
c.connect(('127.0.0.1', port)) # -> ConnectionRefusedError
EOF
podman run --rm --runtime=krun \
--annotation krun.ram_mib=1024 --annotation krun.cpus=2 \
--mount type=bind,src=$PWD/t.py,target=/t.py \
python:3.12-slim python3 /t.pyOutput:
bound: 43859
ConnectionRefusedError: [Errno 111] Connection refusedControl: the same script with a fixed port
(bind(('127.0.0.1', 45677)), connect to 45677) succeeds — fixed-port binds
do not diverge, which confirms the mechanism.
Direct proof of the divergence
Run the same ephemeral listener inside a --network host krun container and
inspect the host side while the guest sleeps on accept():
guest: bound 127.0.0.1:53603 (X)
host: ss -ltn → LISTEN 127.0.0.1:45551 (Y — assigned by the VMM's netns)
connect to X → refused; connect to Y → accepted, guest accept() firesReal-world impact
debugpy (python -m debugpy --listen 0.0.0.0:5678 …) always creates a
127.0.0.1:0 endpoints listener whose adapter child connects back, so every
debugpy session dies after 30 s with
RuntimeError: timed out waiting for adapter to connect. Reproduced in a
bare krun container with debugpy alone. Anything else using ephemeral
listeners is similarly affected.
Suggested fix directions
Return the actually-bound address/port in TsiListenRsp and have the guest
update the socket's local address; or have the guest request a concrete port
for the shadow bind (it already knows X). We currently carry a userspace
workaround (LD_PRELOAD rewriting port-0 binds to concrete ports) and are
happy to contribute a VMM/kernel fix if this direction sounds right.
Related: #819 — same investigation, TSI locking bug
Activity
3I think I have a working patch to fix this issue for the included 6.12.91 kernel. I'm going to revisit it on a fresh mind tomorrow, and if I still think it makes sense I will make a PR against libkrunfw.
I think this explains failures I've been seeing with omp coding agent running inside a podman container with krun as runtime. I see ECONNREFUSED errors when a generated script tries to connect to the PI_TOOL_BRIDGE_* listener it spawns.
Switching podman to passt does cure it, but it breaks the default route init when booting the container, if wireguard is in use. podman doesn't currently let one pass -i <interface> and -g <gateway> to override the auto config, so, I just pass in the gw ip via env var and the login script does the ip config.
Follow-up: for anyone hitting this, booting with
--annotation krun.use_passt=1(real virtio-net NIC, TSI fully disabled) makes the symptom disappear — the samet.pysucceeds with no other changes, which also confirms the bug is TSI-specific. We run our nested-podman workload on passt now.