Guidance needed: increasing SQEs per io_uring_enter for a request/response client workload

Issues/aerospike/aerospike-client-java/#674

Guidance needed: increasing SQEs per io_uring_enter for a request/response client workload

Open

Summary

We migrated an Aerospike client I/O path onto the core netty-transport-native-io_uring transport, hoping to reduce syscalls versus epoll. Functionally it works well. However, the vast majority of our io_uring_enter calls carry exactly 1 SQE, so the syscall-per-operation ratio is close to 1:1 and we have not realised the batching benefit we were hoping for.

We believe we understand why from reading the Netty source (analysis below), but we would appreciate a sanity check on that understanding, and guidance on whether there is any supported way to increase submission batching for this kind of workload — or whether this is simply the expected floor and we should stop looking.

Environment


Netty	4.2.11.Final (`netty-transport-classes-io_uring` / `netty-transport-native-io_uring`)
JDK	25
Platform	Linux x86_64, containerised
Client library	`com.aerospike:aerospike-client-jdk21:10.4.0`
Ring size	default (128)
Buffer ring	tried both enabled and disabled (see below)

Event loop group is constructed as:

new MultiThreadIoEventLoopGroup(threads, IoUringIoHandler.newFactory())

and handed to Aerospike's NettyEventLoops, which round-robins commands across the loops.

Workload shape

This is a latency-sensitive ad-serving service. The relevant characteristics:

~600 QPS of user-profile lookups per pod (mean inter-arrival ~1.67 ms).
Each lookup is an Aerospike batch read of up to 6 keys.
The Aerospike client splits those keys by server node, producing one command (one socket write) per distinct node touched — typically 5-7 commands per request.
Responses arrive independently from each of those 5-7 nodes, at different times, on different sockets.
Requests originate from a Jetty thread pool (thread-per-request), so calls into the event loop always come from outside the event loop thread.

What we observe

strace on the event loop thread, steady state:

io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 6, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 6
io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 1, 1, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1

Roughly 3150 enters/sec on one loop. The distribution is dominated by = 1, with occasional = 5/= 6/= 7 groups. Depth never approaches the 128-entry ring.

Our reading of this, per request:

phase	enters	SQEs each
write fan-out to 5-7 nodes	1	5-7
responses (one per node, independent arrival)	~6	1
total	~7	~1.7 avg

600 QPS × ~7 ≈ 4200 enters/sec, which is the right order of magnitude for what we measure.

Our understanding of the mechanism

From reading 4.2.11 sources — please correct us if any of this is wrong:

The write side already batches optimally. The Aerospike client dispatches all per-node commands for one request back-to-back from the calling thread. Only the first execute() writes the eventfd, because IoUringIoHandler#wakeup coalesces via eventfdAsyncNotify.getAndSet(true). By the time the loop wakes, all 5-7 tasks are queued, runAllTasks drains them together, and we get a single multi-SQE enter. This is the = 6 line above and it is exactly the behaviour we want.
The read side cannot batch, structurally. Each response is a separate CQE arriving at a separate time from a separate socket. Handling it enqueues a re-arm SQE, and then in processCompletionsAndHandleOverflow the next pass sees p == 0 with needSubmit() true and calls submitAndClearNow0 → submitAndGetNow() → submitAndGet0(0) → submit(1, 0, IORING_ENTER_GETEVENTS). That is one enter carrying one SQE, per response. These are ~85% of our enters.
Ring size is irrelevant here. SubmissionQueue#enqueueSqe only force-submits at pending == ringEntries, but IoUringIoHandler#run submits at the end of every loop iteration regardless of depth. So the batch size per enter is "whatever accumulated during one iteration", which at our duty cycle is ~1. To fill 128 entries we estimate we'd need something north of 400k QPS on a single loop.

In other words: there appears to be no "wait until N SQEs are pending before entering" mode, and with independent response arrivals there is nothing to accumulate anyway.

What we have already tried (and what happened)

change	result
Buffer ring (`IORING_REGISTER_BUF_RING`, size 256 × 16 KiB, `largeAllocation`)	No reduction in enters; the `= 5`/`= 6` groups actually increased slightly. Reverted.
`-Dio.netty.iouring.recvMultiShotEnabled=false`	Investigated because Aerospike's `putConnection()` → `clearRead()` → `cancelOutstandingReads()` submits an extra `ASYNC_CANCEL` SQE + produces an `ECANCELED` completion when multishot leaves `readId != 0`. Did not pursue to a conclusion.
Event loop `threads`: 3 → 1 (funnel all Aerospike traffic onto one ring)	No change in SQEs/enter, and total enters/sec across the process stayed flat. Our hypothesis — that round-robin across 3 loops was splitting batchable work — was wrong: the loop is idle between requests, so there is no concurrent backlog to merge regardless of loop count. Reverted.

Questions

Is our analysis correct, particularly point 2 — that per-response single-SQE enters are inherent to a request/response client workload with independent completion arrivals, and not something we've misconfigured?
Is there any supported way to defer or coalesce submissions? Something equivalent to "don't call io_uring_enter until either N SQEs are pending or T microseconds have elapsed". We could not find one. Is such a knob something Netty would consider, or is it deliberately avoided because of the latency cost?
IORING_SETUP_SQPOLL — we note it is not implemented (no IORING_SETUP_SQPOLL constant in Native, no IORING_ENTER_SQ_WAKEUP, though IORING_SQ_NEED_WAKEUP is defined at Native.java:257 but never read). Is that a deliberate design decision? Our own assessment is that it would be a poor fit for us — a kernel polling thread spinning per ring against a loop that is idle ~97% of the time would trade syscalls for a dedicated core, which is the wrong direction since our actual goal is reducing CPU. But we'd value your view on whether SQPOLL is on the roadmap and what workload shape you'd consider it appropriate for.
IORING_SETUP_DEFER_TASKRUN — we understand this is enabled automatically via Native#setupFlags when singleIssuer is true (the default) and the kernel supports it. We are adding startup logging of IoUring.featureString() to confirm it is actually active on our kernel. Are there other setup flags or system properties you'd recommend for a many-small-request client workload like this?
Is ~1 syscall per network operation simply the expected floor here, with io_uring's benefit for our shape being reduced per-syscall cost and better completion handling rather than fewer syscalls? If so we're happy to accept that — we'd just like to stop chasing it. A clear "yes, that's expected" is a genuinely useful answer.

Non-goals / things we know about

We are aware we could add an application-level coalescing layer — buffering independent requests for a small time window and merging their keys into one Aerospike batch, so that N requests hitting the same node collapse into one command. This would genuinely help (the client merges keys per node, so command count is bounded by cluster size rather than key count). We're keeping it as a fallback since it trades latency and adds real complexity. This issue is specifically about whether anything at the Netty/io_uring layer can help first.

Thanks very much for the transport work — happy to provide more traces, run experiments, or test patches if that would be useful.

Activity

mirzakaracic1mo ago

👋 @spartan-29 Thanks for the writeup — there was enough detail in it that we could actually reproduce thi. We stood up a 6-node cluster ran your workload shape (batch read of 6 keys, ~600 TPS, single event loop), patched some counters into Netty's SubmissionQueue so we could count SQEs by opcode, and measured client CPU across a few configurations.

Short answer: your analysis is righ but we think syscall count is the wrong thing to be optimising. There's a much bigger lever, and annoyingly it's the one you listed as a non-goal.

What we found, roughly in order of how much it matters:

Cutting command count cuts client CPU by ~44%. Cutting syscall count by 27% did nothing we could measure.
Around 90% of the commands your batch read issues aren't batch commands at all, they're plain single-record gets.
Neither of your two experiments could have worked. There are specific reasons for both.
Your point 2 needs a small correction: the read side can batch.

Before the numbers, the usual caveats. We ran Netty 4.2.15 (you're on 4.2.11), x86_64, kernel 6.17, and the client and servers were sharing a 2-vCPU box, 30s runs with JIT warmup inside the window. The SQE and syscall counts are exact. The CPU numbers are noisier and we've included the run-to-run spread so you can judge them.

Where the CPU actually goes

Coalescing N requests of k keys is the same thing as one request of N*k keys so we held the key rate fixed at 3,600 keys/s and varied batch size. Every row below does the same amount of useful work:

batch size	request rate	client CPU-seconds (30s)	io_uring_enter
6	600/s	12.34	125,747
12	300/s	10.08	76,756
24	150/s	8.40	42,531
48	75/s	6.90	21,476

That's 44% less CPU at 8x coalescing. It's monotonic across all four points and the 5.4 CPU-second gap is about 6x our run-to-run spread (0.36-0.85), so we're fairly confident it's real.

Compare that with the transport-layer result further down: we got io_uring_enter down 27% while holding command count constant, and CPU didn't budge. Cut command count instead and you get 44%.

So the lever is command count, not syscall count. Every command drags along a send, a recv, command encoding, a buffer allocation, a listener callback and a state machine transition. The syscall around all that is comparatively cheap.

Two things to be careful about. This measures the effect of coalescing, not the cost — a real implementation needs a buffer, a merge step, and a delay while the window fills, so treat 44% as a ceiling on the client-side saving. It also says nothing about the latency you'd be trading away. And as noted, our box is small and shared. But the direction and size are well outside our noise, and it's the same mechanism you described in your non-goals section.

Why there's that much headroom

BatchNodeList.generate groups keys strictly by owning node, one command per node, and never splits by key count. But the call site issues an AsyncBatchSingle.Get — a plain single-record get, not a batch command — any time a node ends up holding exactly one key of the batch (AerospikeClient.java:2216-2221).

With a handful of keys spread across a lot of nodes that's the normal case, not an edge case. Measured on our 6-node cluster, and it lines up with the obvious model (k keys over n nodes) to within 0.3%:

batch size	measured single-record share	predicted
2	91.0%	90.9%
3	82.2%	82.4%
6	60.4%	60.4%
12	30.4%	30.3%

For 6 keys the share climbs with cluster size: 60% at 6 nodes, 88% at 20, 95% at 50. You're seeing 5-7 commands for up to 6 keys, which puts you well north of 20 nodes, so you're probably around 90% single-record gets. Which is also why you get 5-7 responses arriving independently.

Coalescing collapses that. More keys per node means those turn back into real multi-key batch commands, and command count ends up bounded by cluster size rather than key count.

On your analysis

Points 1 and 3 hold up. Point 2 needs a fix though — the read side isn't structurally unable to batch. CompletionQueue#process drains every available CQE in one call, submit0 only enqueues, and enqueueSqe only force-submits at pending == ringEntries. So N completions drained in one pass go out as N SQEs in a single enter.

Same code, same workload, only concurrency changed:

	1 command in flight	200 in flight
avg SQEs per `io_uring_enter`	1.58	82.5
enters carrying exactly 1 SQE	69%	0.06%

Your = 1 is a duty-cycle thing — the loop is idle ~97% of the time and wakes once per arrival. Your conclusion is still right, it's just "nothing to coalesce" rather than "can't coalesce".

Why the two experiments did nothing

recvMultiShotEnabled=false was a no-op for you. Multishot recv only exists in scheduleReadProviderBuffer, which is the buffer-ring path. The non-buffer-ring branch of scheduleRead0 never sets IORING_RECV_MULTISHOT so without a ring there's nothing for the flag to turn off.

The buffer ring is the more interesting one: it was never used. Netty only uses a buffer ring on a channel that has IO_URING_BUFFER_GROUP_ID set. It defaults to -1 and our client never sets it — the only channel options we set are SO_KEEPALIVE, TCP_KEEP*, TCP_NODELAY and AUTO_READ. So registering a ring on IoUringIoHandlerConfig allocates it, registers it with the kernel, and then no channel ever selects it. We ran your config against a control:

30s, same workload	no ring	ring registered (your config)
io_uring_enter	2,013,724	1,998,553
SQEs	3,170,985	3,186,129
SEND / RECV	1,278,327 / 1,382,732	1,279,322 / 1,383,838
POLL_ADD	12	12

Identical. The ring sat there unused and the client stayed on its normal path: a multishot POLL_ADD(POLLIN) armed once per connection (that's the 12, against 1.38M RECVs), then one RECV SQE per response, one enter, no cancel. Which is your = 1 line.

For completeness we did push the transport layer as far as it goes, and it doesn't pay. You need two changes together: a buffer ring to get Netty onto multishot recv and dropping our per-command setAutoRead(false) so the multishot doesn't get cancelled after every completion. Either one alone is useless or actively worse (a ring on its own costs an extra ASYNC_CANCEL SQE per command). Together, at 600 TPS over 30s:

	baseline	ring + autoRead fix
io_uring_enter	131,845	96,712 (-27%)
SQEs	211,238	132,419 (-37%)
RECV SQEs	75,329	6

RECV goes from one per response to one per connection. And the CPU 3 runs each:

	mean CPU-s	run-to-run range
epoll	12.28	0.85
io_uring	11.94	0.66
io_uring + ring + autoRead fix	12.47	0.69

Nothing. The spread between the three arms is smaller than the spread between repeats of the same arm. epoll and io_uring are indistinguishable at this rate too.

Your questions

Correct, apart from point 2 above. Nothing misconfigured on your end.
No, there's no such knob. It also wouldn't help — at 1.67ms inter-arrival, any "wait for N SQEs or T microseconds" timer would eat your latency budget before it had anything to merge.
Not implemented, and there's some useful discussion here: https://github.com/netty/netty/issues/15199
Nothing left to set. Everything worthwhile is already negotiated in Native#setupFlags. The one knob defaulted off is recvsendBundleEnabled and that needs a buffer ring, so per the section above it wouldn't move your syscall count either. Kernel version matters a lot more than flags here.
Yes, ~1 syscall per response is the floor the way the client currently drives reads. And we'd go a bit further than you asked: even the extra 27% we could squeeze out didn't show up in CPU at all. If CPU is what you're actually chasing, the first section is where to look.

If your own profiling disagrees with any of this we'd genuinely like to see it — our box is small and it's possible something shows up at your scale that didn't at ours.