#814
fix: CLIENT-5768 prevent a hung socket close from stalling tend and the connection pool
Summary
A socket close that never returns could turn into a full client outage:
singleConnectionHeap.DropIdleTailclosed idle connections on the tend goroutine while holding the heap shard lock. A blockednet.Conn.Close()held that lock, so everyPollandOfferon the shard, and every command to that node, waited behind it.Cluster.tendwaits for every node'sRefresh, so the tend cycle for the whole cluster stalled too. The error rate counters were never reset, and the other nodes eventually failed withMAX_ERROR_RATE.
Changes
- Add
Connection.closeAsync(). It releases the connection right away (connection count, stats, data buffer) and closes only the underlying socket in a separate goroutine.DropIdleTailnow uses it, so the tend goroutine and the heap lock never wait on a socket close. singleConnectionHeap.cleanupnow empties the heap under the lock and closes the connections after releasing it.- Add regression tests that use a fake
net.ConnwhoseCloseblocks. Both tests fail without this change.
Node removal (Node.Close) still closes sockets synchronously, because Client.Close() shares that path and callers expect connections to be closed when it returns.
Testing
connection_heap_test.go, plus the node and connection specs, pass with-raceagainst a local server.- On the full integration suite with
-race, this branch has the same results as the base branch when both run with the same random seed. With some spec orders, both show races and failures in the load, truncate, empty bin name, andExpErrorDetailspecs. Those come from command buffers shared under concurrent load and are unrelated to this change.
Activity
3dmishra-aeroreviewed6d ago
Use Branch name with Jira-Id-short-Title