#934

TSI UDP: recvfrom()/recvmsg() on an unconnected socket reports the last sendto() destination, not the real sender

Opendjbclarkopened 14d ago

Summary

On a TSI (virtio-vsock) guest, an unconnected SOCK_DGRAM socket that talks to more than one peer gets the wrong source address from recvfrom()/recvmsg(): the kernel returns the destination of the most recent sendto() on that socket, whoever actually sent the datagram. Protocols that multiplex peers on one UDP socket and match replies by source address (for example OpenAFS Rx) break or misattribute replies.

Evidence

  1. Guest kernel, libkrunfw (libkrun/libkrunfw) patches/0011-Transparent-Socket-Impersonation-implementation.patch, tsi_dgram_recvmsg(): after the vsock recvmsg, memcpy(msg->msg_name, tsk->sendto_addr, tsk->sendto_addr_len). tsk->sendto_addr is overwritten by every tsi_dgram_sendmsg() with a msg_name.
  2. Host VMM, libkrun src/devices/src/virtio/vsock/tsi_dgram/unix.rs, recv_to_pkt(): uses recv(), so the real sender is dropped before the datagram reaches the guest (no field for it in the VSOCK_OP_RW packet). TsiDgramProxy.sendto_addr is likewise a single address per proxy.
  3. Reproduction: the script below (stdlib only; also at https://forgejo.mit.edu/djbclark/get/src/branch/main/docs/experiments/udp-source-test.py), two DNS servers A and B:
    A then B: reply to query 1 (sent to 18.0.70.160) reported from 18.0.72.3 BAD
    A then B: reply to query 2 (sent to 18.0.72.3) reported from 18.0.72.3 ok
    B then A: reply to query 3 (sent to 18.0.72.3) reported from 18.0.70.160 BAD
    Run on a Hyades microVM (libkrun TSI, guest kernel 6.12.109) on 2026-10-16. A single-peer socket is fine.
  4. Consequence seen: OpenAFS afsd.fuse (one Rx socket for the VLDB on 7003 and fileservers on 7000) mounts but ls /afs/athena.mit.edu never completes; replies are matched to the wrong peer and the requests are retransmitted forever.
  5. No existing issue found (searched libkrun and libkrunfw issues and PRs for recvfrom, source address, sendto_addr, tsi_dgram_recvmsg; Jina, Parallel and Exa web search). Related but different: libkrunfw#132 (tsi_connect commits to S_INET too early with SOCK_DGRAM), PR 659 (sendto_addr bind).

Expected / fix direction

The VMM knows the true sender (recvfrom() on the host socket). Carry it in the datagram to the guest (e.g. a small sockaddr header before the payload, gated by a feature bit so old kernels still work) and have tsi_dgram_recvmsg copy that into msg_name, instead of tsk->sendto_addr. Until then document that multi-peer UDP on one socket is unsupported under TSI.

Workarounds

  1. One connected socket per peer in the application (works: the kernel's getpeername/connected path is exact for a single peer).
  2. virtio-net with passt/gvproxy instead of TSI (not available on Hyades).

Reproduction script

import socket
import struct
import sys

A = (sys.argv[1] if len(sys.argv) > 1 else "18.0.70.160", 53)
B = (sys.argv[2] if len(sys.argv) > 2 else "18.0.72.3", 53)


def query(qid):
    return (struct.pack(">HHHHHH", qid, 0x0100, 1, 0, 0, 0)
            + b"\x07example\x03com\x00" + struct.pack(">HH", 1, 1))


def run(label, seq):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.settimeout(4)
    for qid, dst in seq:
        s.sendto(query(qid), dst)
    sent = {qid: dst[0] for qid, dst in seq}
    for _ in seq:
        data, src = s.recvfrom(1500)
        qid = struct.unpack(">H", data[:2])[0]
        ok = "ok " if src[0] == sent[qid] else "BAD"
        print(f"{label}: reply to query {qid} (sent to {sent[qid]}) reported from {src[0]} {ok}")


run("A then B", [(1, A), (2, B)])
run("B then A", [(3, B), (4, A)])

Found while trying to run the OpenAFS FUSE client in a libkrun guest (Hyades, MIT SIPB). Happy to test or contribute a fix if the direction above (carry the real sender in the datagram, gated by a feature bit) is acceptable.

Activity

0

No activity yet.