#11529

Tool order depends on registration order, so the same tools can bust the prompt cache

OpenDiegoGBrisaopened 11d ago

What happened?

getCurrentTools() in utils/transcript.ts returns tools in insertion order, and a tool that is removed and added again goes to the end of the Map. So when tools show up after the session starts (MCP servers connecting or reconnecting, tool_search loading tools, extensions registering late), the final order depends on who finished first. Same tool set, different bytes in the request, and the provider writes the whole cache again.

On providers without supportsMidConvoToolChanges (Bedrock for example) every one of those is a full cache write, and with a big prefix that gets expensive. I hit it a lot with 3-4 MCP servers that connect at different speeds.

Steps to reproduce

Quick way to see it without MCP:

const a = getCurrentTools([sys({ toolsAdded: [x, y] }), sys({ toolsRemoved: [x] }), sys({ toolsAdded: [x] })])
const b = getCurrentTools([sys({ toolsAdded: [x, y] })])
// a = [y, x], b = [x, y]: same tools, different request

In a real session: two direct MCP servers that take different times to connect, run a few sessions on Bedrock, and compare the tools array of the requests. The order changes between runs and cacheRead drops to 0 on the request where it changed.

Expected behavior

The same tool set always serializes the same way, no matter in which order the tools arrived. Sorting the resolved list by name would do it (one cache miss when you upgrade, then stable).

Version

1.4.0, same on main (cc9c52310)

Happy to send a PR for this one too.

Activity

0

No activity yet.