#682

socketTimeout bounds each socket read rather than the attempt, so a slow-feeding server can keep a command alive past totalTimeout

Openaltaiezioropened 1mo ago

Summary

totalTimeout is enforced between retries but not within a single attempt. Inside an attempt, reads are bounded individually by socketTimeout. A peer that emits a little data before each socketTimeout expiry therefore keeps one attempt alive indefinitely, and totalTimeout never fires.

Version

com.aerospike:aerospike-client-jdk21 10.4.0, and current stage (7577294e). The same code is on stage-jdk8, so this is not specific to the Java 21 line.

Detail

Connection.readFully:

while (pos < length) {
    int count = in.read(buffer, pos, length - pos);
    if (count < 0) throw new EOFException();
    pos += count;
}

SO_TIMEOUT applies to each read(), not to the loop. SyncCommand.executeCommand consults deadline only after an attempt fails, in the retry block. MultiCommand streams an entire batch or scan response through this loop, so the exposure grows with response size.

Reproducer

No Aerospike server required — Connection is a socket wrapper, so a plain ServerSocket that drips bytes is enough. One byte every 250 ms against socketTimeout = 500, requesting more bytes than the drip can deliver:

Connection conn = new Connection(dripServerAddress, 500);
conn.readFully(new byte[10_000], 10_000);

Observed: still running after 3,000 ms, with no single read ever reaching socketTimeout. It does not terminate on its own.

readFully was still running after 3000ms with socketTimeout=500ms. A byte arriving
every 250ms resets the per-read timeout, and nothing bounds the attempt as a whole,
so the command outlives any totalTimeout the caller configured.
Full JUnit test (needs no server)
/*
 * Copyright 2012-2026 Aerospike, Inc.
 *
 * Portions may be licensed to Aerospike, Inc. under one or more contributor
 * license agreements WHICH ARE COMPATIBLE WITH THE APACHE LICENSE, VERSION 2.0.
 *
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not
 * use this file except in compliance with the License. You may obtain a copy of
 * the License at http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under
 * the License.
 */
package com.aerospike.client;

import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;

import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import com.aerospike.client.cluster.Connection;

/**
 * socketTimeout is applied to each socket read rather than to the attempt as a whole.
 * Connection.readFully loops on in.read(...) and only that individual call is bounded, so a peer
 * that emits a little data before each socketTimeout expiry keeps the loop running. SyncCommand
 * consults its deadline only between retries, so totalTimeout never fires for such an attempt.
 * <p>
 * This test needs no Aerospike server. Connection is a socket wrapper, so a plain ServerSocket
 * that drips bytes is enough to hold readFully open indefinitely.
 */
public class ConnectionReadDeadlineTest {
	/** Bounds each individual read. No single read ever reaches it. */
	private static final int SOCKET_TIMEOUT_MILLIS = 500;

	/** Comfortably below SOCKET_TIMEOUT_MILLIS, so every read returns a byte instead of timing out. */
	private static final long DRIP_INTERVAL_MILLIS = 250;

	/** Far more than the drip can deliver, so the read loop cannot complete on its own. */
	private static final int BYTES_REQUESTED = 10_000;

	/** How long the caller is given before the read is declared unbounded. */
	private static final long OBSERVATION_MILLIS = 3_000;

	private ServerSocket server;
	private Thread dripper;
	private volatile boolean running = true;

	@Before
	public void startDripServer() throws Exception {
		server = new ServerSocket(0, 1);

		dripper = new Thread(() -> {
			try (Socket accepted = server.accept()) {
				OutputStream out = accepted.getOutputStream();

				while (running) {
					out.write(0);
					out.flush();
					Thread.sleep(DRIP_INTERVAL_MILLIS);
				}
			}
			catch (Exception e) {
				// The test closes the socket underneath this thread when it finishes.
			}
		});
		dripper.setDaemon(true);
		dripper.start();
	}

	@After
	public void stopDripServer() throws Exception {
		running = false;

		if (dripper != null) {
			dripper.interrupt();
		}

		if (server != null) {
			server.close();
		}
	}

	@Test
	public void readIsBoundedInTotalAndNotOnlyPerRead() throws Exception {
		InetSocketAddress address = new InetSocketAddress("127.0.0.1", server.getLocalPort());
		Connection conn = new Connection(address, SOCKET_TIMEOUT_MILLIS);

		CountDownLatch finished = new CountDownLatch(1);
		AtomicReference<Throwable> thrown = new AtomicReference<>();

		Thread caller = new Thread(() -> {
			try {
				conn.readFully(new byte[BYTES_REQUESTED], BYTES_REQUESTED);
			}
			catch (Throwable t) {
				thrown.set(t);
			}
			finally {
				finished.countDown();
			}
		});
		caller.setDaemon(true);
		caller.start();

		boolean returned = finished.await(OBSERVATION_MILLIS, TimeUnit.MILLISECONDS);

		try {
			if (! returned) {
				fail("readFully was still running after " + OBSERVATION_MILLIS + "ms with socketTimeout="
					+ SOCKET_TIMEOUT_MILLIS + "ms. A byte arriving every " + DRIP_INTERVAL_MILLIS
					+ "ms resets the per-read timeout, and nothing bounds the attempt as a whole, "
					+ "so the command outlives any totalTimeout the caller configured.");
			}

			assertTrue("readFully should have reported a timeout, but it returned normally",
				thrown.get() != null);
		}
		finally {
			caller.interrupt();
			conn.close();
		}
	}
}

I can open this as a PR against stage if useful, though on its own it is a deliberately failing test with no fix behind it.

Impact

A command can exceed its configured totalTimeout without bound. On the sync batch path this also blocks the caller, because that wait is itself unbounded — see #680. The two compound, but they are independent defects and this one is the root cause.

Question before any patch

Adding a cumulative deadline to the read loop interacts with timeoutDelay and ConnectionRecover. Today Connection.ReadTimeout deliberately carries the partial buffer and parse state so the socket can be drained in the background and reused. A new deadline exit has to decide recover-versus-close, and getting that wrong would leak sockets. It also changes timeout semantics for every sync command, not just batch, so it may be a behaviour change you would want to gate or document rather than apply silently.

I would rather hear which shape you want before proposing a patch. If a cumulative bound is unwelcome, documenting that socketTimeout is per-read and totalTimeout is per-retry-sequence would at least remove the surprise.

This report was prepared with AI assistance.

Activity

0

No activity yet.