#814

fix: CLIENT-5768 prevent a hung socket close from stalling tend and the connection pool

Openjuliannguyen4wants to merge intofrom

Summary

A socket close that never returns could turn into a full client outage:

  • singleConnectionHeap.DropIdleTail closed idle connections on the tend goroutine while holding the heap shard lock. A blocked net.Conn.Close() held that lock, so every Poll and Offer on the shard, and every command to that node, waited behind it.
  • Cluster.tend waits for every node's Refresh, so the tend cycle for the whole cluster stalled too. The error rate counters were never reset, and the other nodes eventually failed with MAX_ERROR_RATE.

Changes

  • Add Connection.closeAsync(). It releases the connection right away (connection count, stats, data buffer) and closes only the underlying socket in a separate goroutine. DropIdleTail now uses it, so the tend goroutine and the heap lock never wait on a socket close.
  • singleConnectionHeap.cleanup now empties the heap under the lock and closes the connections after releasing it.
  • Add regression tests that use a fake net.Conn whose Close blocks. Both tests fail without this change.

Node removal (Node.Close) still closes sockets synchronously, because Client.Close() shares that path and callers expect connections to be closed when it returns.

Testing

  • connection_heap_test.go, plus the node and connection specs, pass with -race against a local server.
  • On the full integration suite with -race, this branch has the same results as the base branch when both run with the same random seed. With some spec orders, both show races and failures in the load, truncate, empty bin name, and ExpErrorDetail specs. Those come from command buffers shared under concurrent load and are unrelated to this change.

Activity

3
dmishra-aeroreviewed6d ago
dmishra-aero6d ago

Use Branch name with Jira-Id-short-Title