#17414
Issues/diegosouzapw/OmniRoute/#17414

Stream relay concatenates a restarted `tool_calls[].function.arguments` fragment onto the partial one, producing invalid JSON for the client

Opendiptimansinghopened 1d ago

OmniRoute Version

3.8.52

Installation Method

npm (global)

Operating System

Windows

OS Version

No response

Node.js Version

No response

Provider(s) Involved

No response

Model(s) Involved

No response

Client Tool

No response

Description

Summary

On /v1/chat/completions with stream: true, OmniRoute can emit a tool call whose function.arguments string is the partial buffer followed by a full restart of the same arguments, concatenated instead of replaced. The client receives syntactically invalid JSON and cannot execute the tool, so the turn is lost.

Example shape observed (abbreviated; the second fragment restarts from the opening { with a different key ordering than the first):

{"content": "import json, os, glob\n\nbase = os.path.expanduser(...
{"content": "import json, os, glob\n\nbase = os.path.expanduser(...)   <- full restart, appended

Impact

Any OpenAI-compatible agent client that parses tool arguments strictly fails the call. In the client used here (OpenCode 2.1.2, AI SDK stream parser) it surfaces as:

"The tool input could not be parsed as JSON, so the call was not executed."

The work item is abandoned mid-turn and must be re-issued.

Environment

  • OmniRoute 3.8.52 (global npm install)
  • Combo: mb-deepseek-flash, strategy fill-first
  • Client: OpenCode 2.1.2 (/v1/chat/completions, OpenAI format, streaming)
  • Compression: RTK only, auto-trigger

Evidence

1. The wire is not the logged body

For every sampled failed call, the stored responseBody in ~/.omniroute/call_logs contains valid JSON for that tool call. Validating 1842 response-side argument blobs across a day of logs produced 0 invalid values. So whatever the client received differs from what OmniRoute believes it sent.

2. It is not specific to one provider, model, or combo target

Across one day, 278 requests had the client-side parse error in their history. Distribution by the combo target that served them:

combo target servedrequests
1 (no fallback at all)91
22
3266
49

Failures appear on deepseek-v4.1-flash, openrouter/auto, gemini-3.8-flash-low, grok-4.7, claude-opus-4-6-thinking, claude-sonnet-4-6, and deepseek-ai/deepseek-v4-flash-0731. 91 of them had zero fallbacks, so mid-request failover is not a precondition.

3. It is not compression

Compression runs on the request (compressionStage: "pre-translation" in handlers/chatCore.ts, operating on the request body). In the whole services/compression subtree, .arguments appears exactly once — engines/rtk/index.ts:604, inside a read-only JSON.parse(fn.arguments) for a command lookup. There is no assignment to tool_calls[].function.arguments anywhere. RTK's shouldCompressMessage only ever rewrites message.content for role: "tool" / "assistant". Confirmed not the cause.

Minimal reproduction (of the accumulator behaviour)

Self-contained; exercises the same accumulation contract as open-sse/utils/toolCallArguments.ts:

const append = (current, incoming) => {
  const existing = typeof current === "string" ? current : "";
  const next = typeof incoming === "string" ? incoming : incoming == null ? "" : JSON.stringify(incoming);
  if (!existing) return next;
  if (!next) return existing;
  if (next === existing) return existing;
  if (next.startsWith(existing)) return next;
  return existing + next;
};

const partial = '{"path":"a.ts","newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId';
const restart = '{"newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId FROM dbo.Areas WHERE Id = @areaId;"}';

const out = append(partial, restart);
try { JSON.parse(out); console.log("valid"); }
catch (e) { console.log("INVALID:", e.message); }   // INVALID

restart is a superset of the payload but not a string-prefix of partial, because the key ordering differs. appendToolCallArgumentDelta therefore falls through to existing + next and emits both copies.

Likely cause

open-sse/utils/toolCallArguments.ts already exists specifically to collapse snapshot-style deltas, and its header comment cites the exact duplication hazard. It is wired into translator/response/responsesTransformer.ts, translator/response/openai-to-claude.ts, translator/response/openai-responses.ts, handlers/sseParser.ts, and executors/bedrock.ts — i.e. the translator paths. It does not appear to be applied on the path that relays an already-OpenAI-format upstream stream through to an OpenAI-format client, so a non-conformant upstream that re-sends a full snapshot can reach the client as two concatenated fragments.

Separately, handlers/chatCore/toolCallSpecViolationAudit.ts already audits "duplicate tool_calls entry" as a known upstream contract violation — this looks like the same class of defect one layer earlier, on the arguments string rather than the whole call.

Two candidate contributing behaviours, both worth checking:

  1. A relay that forwards upstream arguments fragments verbatim without the collapse helper.
  2. An abandoned attempt leaving its partial arguments buffer in place, so the next attempt's fragments are appended to it. (Note: comboAttemptLoop.ts resets state.fallbackCount per set-try but the count is incremented elsewhere; the buffer-reset path is the thing to verify — resuming an attempt should discard, not extend, a partial argument buffer.)

Suggested fix

  • Apply appendToolCallArgumentDelta on the client-facing relay path, not just the translator paths (or normalise upstream snapshot-style deltas into incremental deltas before relaying).
  • Reset the per-attempt arguments buffer whenever an attempt is abandoned.
  • Consider a JSON.parse assertion on the assembled arguments before finish_reason: "tool_calls" is emitted, as a guard against both cases.

Note on appendToolCallArgumentDelta itself

The existing next.startsWith(existing) rule is the right conservative choice — its comment correctly warns that fuzzy overlap dedup silently truncates legitimate incremental deltas. The gap here is coverage (where it is applied), not the rule. Key-order-changed snapshots are inherently ambiguous from the fragment alone, so a per-attempt reset is likely the safer lever than widening the string heuristic.

Steps to Reproduce

Summary

On /v1/chat/completions with stream: true, OmniRoute can emit a tool call whose function.arguments string is the partial buffer followed by a full restart of the same arguments, concatenated instead of replaced. The client receives syntactically invalid JSON and cannot execute the tool, so the turn is lost.

Example shape observed (abbreviated; the second fragment restarts from the opening { with a different key ordering than the first):

{"content": "import json, os, glob\n\nbase = os.path.expanduser(...
{"content": "import json, os, glob\n\nbase = os.path.expanduser(...)   <- full restart, appended

Impact

Any OpenAI-compatible agent client that parses tool arguments strictly fails the call. In the client used here (OpenCode 2.1.2, AI SDK stream parser) it surfaces as:

"The tool input could not be parsed as JSON, so the call was not executed."

The work item is abandoned mid-turn and must be re-issued.

Environment

  • OmniRoute 3.8.52 (global npm install)
  • Combo: mb-deepseek-flash, strategy fill-first
  • Client: OpenCode 2.1.2 (/v1/chat/completions, OpenAI format, streaming)
  • Compression: RTK only, auto-trigger

Evidence

1. The wire is not the logged body

For every sampled failed call, the stored responseBody in ~/.omniroute/call_logs contains valid JSON for that tool call. Validating 1842 response-side argument blobs across a day of logs produced 0 invalid values. So whatever the client received differs from what OmniRoute believes it sent.

2. It is not specific to one provider, model, or combo target

Across one day, 278 requests had the client-side parse error in their history. Distribution by the combo target that served them:

combo target servedrequests
1 (no fallback at all)91
22
3266
49

Failures appear on deepseek-v4.1-flash, openrouter/auto, gemini-3.8-flash-low, grok-4.7, claude-opus-4-6-thinking, claude-sonnet-4-6, and deepseek-ai/deepseek-v4-flash-0731. 91 of them had zero fallbacks, so mid-request failover is not a precondition.

3. It is not compression

Compression runs on the request (compressionStage: "pre-translation" in handlers/chatCore.ts, operating on the request body). In the whole services/compression subtree, .arguments appears exactly once — engines/rtk/index.ts:604, inside a read-only JSON.parse(fn.arguments) for a command lookup. There is no assignment to tool_calls[].function.arguments anywhere. RTK's shouldCompressMessage only ever rewrites message.content for role: "tool" / "assistant". Confirmed not the cause.

Minimal reproduction (of the accumulator behaviour)

Self-contained; exercises the same accumulation contract as open-sse/utils/toolCallArguments.ts:

const append = (current, incoming) => {
  const existing = typeof current === "string" ? current : "";
  const next = typeof incoming === "string" ? incoming : incoming == null ? "" : JSON.stringify(incoming);
  if (!existing) return next;
  if (!next) return existing;
  if (next === existing) return existing;
  if (next.startsWith(existing)) return next;
  return existing + next;
};

const partial = '{"path":"a.ts","newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId';
const restart = '{"newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId FROM dbo.Areas WHERE Id = @areaId;"}';

const out = append(partial, restart);
try { JSON.parse(out); console.log("valid"); }
catch (e) { console.log("INVALID:", e.message); }   // INVALID

restart is a superset of the payload but not a string-prefix of partial, because the key ordering differs. appendToolCallArgumentDelta therefore falls through to existing + next and emits both copies.

Likely cause

open-sse/utils/toolCallArguments.ts already exists specifically to collapse snapshot-style deltas, and its header comment cites the exact duplication hazard. It is wired into translator/response/responsesTransformer.ts, translator/response/openai-to-claude.ts, translator/response/openai-responses.ts, handlers/sseParser.ts, and executors/bedrock.ts — i.e. the translator paths. It does not appear to be applied on the path that relays an already-OpenAI-format upstream stream through to an OpenAI-format client, so a non-conformant upstream that re-sends a full snapshot can reach the client as two concatenated fragments.

Separately, handlers/chatCore/toolCallSpecViolationAudit.ts already audits "duplicate tool_calls entry" as a known upstream contract violation — this looks like the same class of defect one layer earlier, on the arguments string rather than the whole call.

Two candidate contributing behaviours, both worth checking:

  1. A relay that forwards upstream arguments fragments verbatim without the collapse helper.
  2. An abandoned attempt leaving its partial arguments buffer in place, so the next attempt's fragments are appended to it. (Note: comboAttemptLoop.ts resets state.fallbackCount per set-try but the count is incremented elsewhere; the buffer-reset path is the thing to verify — resuming an attempt should discard, not extend, a partial argument buffer.)

Suggested fix

  • Apply appendToolCallArgumentDelta on the client-facing relay path, not just the translator paths (or normalise upstream snapshot-style deltas into incremental deltas before relaying).
  • Reset the per-attempt arguments buffer whenever an attempt is abandoned.
  • Consider a JSON.parse assertion on the assembled arguments before finish_reason: "tool_calls" is emitted, as a guard against both cases.

Note on appendToolCallArgumentDelta itself

The existing next.startsWith(existing) rule is the right conservative choice — its comment correctly warns that fuzzy overlap dedup silently truncates legitimate incremental deltas. The gap here is coverage (where it is applied), not the rule. Key-order-changed snapshots are inherently ambiguous from the fragment alone, so a per-attempt reset is likely the safer lever than widening the string heuristic.

Expected Behavior

Summary

On /v1/chat/completions with stream: true, OmniRoute can emit a tool call whose function.arguments string is the partial buffer followed by a full restart of the same arguments, concatenated instead of replaced. The client receives syntactically invalid JSON and cannot execute the tool, so the turn is lost.

Example shape observed (abbreviated; the second fragment restarts from the opening { with a different key ordering than the first):

{"content": "import json, os, glob\n\nbase = os.path.expanduser(...
{"content": "import json, os, glob\n\nbase = os.path.expanduser(...)   <- full restart, appended

Impact

Any OpenAI-compatible agent client that parses tool arguments strictly fails the call. In the client used here (OpenCode 2.1.2, AI SDK stream parser) it surfaces as:

"The tool input could not be parsed as JSON, so the call was not executed."

The work item is abandoned mid-turn and must be re-issued.

Environment

  • OmniRoute 3.8.52 (global npm install)
  • Combo: mb-deepseek-flash, strategy fill-first
  • Client: OpenCode 2.1.2 (/v1/chat/completions, OpenAI format, streaming)
  • Compression: RTK only, auto-trigger

Evidence

1. The wire is not the logged body

For every sampled failed call, the stored responseBody in ~/.omniroute/call_logs contains valid JSON for that tool call. Validating 1842 response-side argument blobs across a day of logs produced 0 invalid values. So whatever the client received differs from what OmniRoute believes it sent.

2. It is not specific to one provider, model, or combo target

Across one day, 278 requests had the client-side parse error in their history. Distribution by the combo target that served them:

combo target servedrequests
1 (no fallback at all)91
22
3266
49

Failures appear on deepseek-v4.1-flash, openrouter/auto, gemini-3.8-flash-low, grok-4.7, claude-opus-4-6-thinking, claude-sonnet-4-6, and deepseek-ai/deepseek-v4-flash-0731. 91 of them had zero fallbacks, so mid-request failover is not a precondition.

3. It is not compression

Compression runs on the request (compressionStage: "pre-translation" in handlers/chatCore.ts, operating on the request body). In the whole services/compression subtree, .arguments appears exactly once — engines/rtk/index.ts:604, inside a read-only JSON.parse(fn.arguments) for a command lookup. There is no assignment to tool_calls[].function.arguments anywhere. RTK's shouldCompressMessage only ever rewrites message.content for role: "tool" / "assistant". Confirmed not the cause.

Minimal reproduction (of the accumulator behaviour)

Self-contained; exercises the same accumulation contract as open-sse/utils/toolCallArguments.ts:

const append = (current, incoming) => {
  const existing = typeof current === "string" ? current : "";
  const next = typeof incoming === "string" ? incoming : incoming == null ? "" : JSON.stringify(incoming);
  if (!existing) return next;
  if (!next) return existing;
  if (next === existing) return existing;
  if (next.startsWith(existing)) return next;
  return existing + next;
};

const partial = '{"path":"a.ts","newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId';
const restart = '{"newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId FROM dbo.Areas WHERE Id = @areaId;"}';

const out = append(partial, restart);
try { JSON.parse(out); console.log("valid"); }
catch (e) { console.log("INVALID:", e.message); }   // INVALID

restart is a superset of the payload but not a string-prefix of partial, because the key ordering differs. appendToolCallArgumentDelta therefore falls through to existing + next and emits both copies.

Likely cause

open-sse/utils/toolCallArguments.ts already exists specifically to collapse snapshot-style deltas, and its header comment cites the exact duplication hazard. It is wired into translator/response/responsesTransformer.ts, translator/response/openai-to-claude.ts, translator/response/openai-responses.ts, handlers/sseParser.ts, and executors/bedrock.ts — i.e. the translator paths. It does not appear to be applied on the path that relays an already-OpenAI-format upstream stream through to an OpenAI-format client, so a non-conformant upstream that re-sends a full snapshot can reach the client as two concatenated fragments.

Separately, handlers/chatCore/toolCallSpecViolationAudit.ts already audits "duplicate tool_calls entry" as a known upstream contract violation — this looks like the same class of defect one layer earlier, on the arguments string rather than the whole call.

Two candidate contributing behaviours, both worth checking:

  1. A relay that forwards upstream arguments fragments verbatim without the collapse helper.
  2. An abandoned attempt leaving its partial arguments buffer in place, so the next attempt's fragments are appended to it. (Note: comboAttemptLoop.ts resets state.fallbackCount per set-try but the count is incremented elsewhere; the buffer-reset path is the thing to verify — resuming an attempt should discard, not extend, a partial argument buffer.)

Suggested fix

  • Apply appendToolCallArgumentDelta on the client-facing relay path, not just the translator paths (or normalise upstream snapshot-style deltas into incremental deltas before relaying).
  • Reset the per-attempt arguments buffer whenever an attempt is abandoned.
  • Consider a JSON.parse assertion on the assembled arguments before finish_reason: "tool_calls" is emitted, as a guard against both cases.

Note on appendToolCallArgumentDelta itself

The existing next.startsWith(existing) rule is the right conservative choice — its comment correctly warns that fuzzy overlap dedup silently truncates legitimate incremental deltas. The gap here is coverage (where it is applied), not the rule. Key-order-changed snapshots are inherently ambiguous from the fragment alone, so a per-attempt reset is likely the safer lever than widening the string heuristic.

Actual Behavior

Summary

On /v1/chat/completions with stream: true, OmniRoute can emit a tool call whose function.arguments string is the partial buffer followed by a full restart of the same arguments, concatenated instead of replaced. The client receives syntactically invalid JSON and cannot execute the tool, so the turn is lost.

Example shape observed (abbreviated; the second fragment restarts from the opening { with a different key ordering than the first):

{"content": "import json, os, glob\n\nbase = os.path.expanduser(...
{"content": "import json, os, glob\n\nbase = os.path.expanduser(...)   <- full restart, appended

Impact

Any OpenAI-compatible agent client that parses tool arguments strictly fails the call. In the client used here (OpenCode 2.1.2, AI SDK stream parser) it surfaces as:

"The tool input could not be parsed as JSON, so the call was not executed."

The work item is abandoned mid-turn and must be re-issued.

Environment

  • OmniRoute 3.8.52 (global npm install)
  • Combo: mb-deepseek-flash, strategy fill-first
  • Client: OpenCode 2.1.2 (/v1/chat/completions, OpenAI format, streaming)
  • Compression: RTK only, auto-trigger

Evidence

1. The wire is not the logged body

For every sampled failed call, the stored responseBody in ~/.omniroute/call_logs contains valid JSON for that tool call. Validating 1842 response-side argument blobs across a day of logs produced 0 invalid values. So whatever the client received differs from what OmniRoute believes it sent.

2. It is not specific to one provider, model, or combo target

Across one day, 278 requests had the client-side parse error in their history. Distribution by the combo target that served them:

combo target servedrequests
1 (no fallback at all)91
22
3266
49

Failures appear on deepseek-v4.1-flash, openrouter/auto, gemini-3.8-flash-low, grok-4.7, claude-opus-4-6-thinking, claude-sonnet-4-6, and deepseek-ai/deepseek-v4-flash-0731. 91 of them had zero fallbacks, so mid-request failover is not a precondition.

3. It is not compression

Compression runs on the request (compressionStage: "pre-translation" in handlers/chatCore.ts, operating on the request body). In the whole services/compression subtree, .arguments appears exactly once — engines/rtk/index.ts:604, inside a read-only JSON.parse(fn.arguments) for a command lookup. There is no assignment to tool_calls[].function.arguments anywhere. RTK's shouldCompressMessage only ever rewrites message.content for role: "tool" / "assistant". Confirmed not the cause.

Minimal reproduction (of the accumulator behaviour)

Self-contained; exercises the same accumulation contract as open-sse/utils/toolCallArguments.ts:

const append = (current, incoming) => {
  const existing = typeof current === "string" ? current : "";
  const next = typeof incoming === "string" ? incoming : incoming == null ? "" : JSON.stringify(incoming);
  if (!existing) return next;
  if (!next) return existing;
  if (next === existing) return existing;
  if (next.startsWith(existing)) return next;
  return existing + next;
};

const partial = '{"path":"a.ts","newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId';
const restart = '{"newString":"        IF @areaId IS NOT NULL SELECT @hqId = HeadQuarterId FROM dbo.Areas WHERE Id = @areaId;"}';

const out = append(partial, restart);
try { JSON.parse(out); console.log("valid"); }
catch (e) { console.log("INVALID:", e.message); }   // INVALID

restart is a superset of the payload but not a string-prefix of partial, because the key ordering differs. appendToolCallArgumentDelta therefore falls through to existing + next and emits both copies.

Likely cause

open-sse/utils/toolCallArguments.ts already exists specifically to collapse snapshot-style deltas, and its header comment cites the exact duplication hazard. It is wired into translator/response/responsesTransformer.ts, translator/response/openai-to-claude.ts, translator/response/openai-responses.ts, handlers/sseParser.ts, and executors/bedrock.ts — i.e. the translator paths. It does not appear to be applied on the path that relays an already-OpenAI-format upstream stream through to an OpenAI-format client, so a non-conformant upstream that re-sends a full snapshot can reach the client as two concatenated fragments.

Separately, handlers/chatCore/toolCallSpecViolationAudit.ts already audits "duplicate tool_calls entry" as a known upstream contract violation — this looks like the same class of defect one layer earlier, on the arguments string rather than the whole call.

Two candidate contributing behaviours, both worth checking:

  1. A relay that forwards upstream arguments fragments verbatim without the collapse helper.
  2. An abandoned attempt leaving its partial arguments buffer in place, so the next attempt's fragments are appended to it. (Note: comboAttemptLoop.ts resets state.fallbackCount per set-try but the count is incremented elsewhere; the buffer-reset path is the thing to verify — resuming an attempt should discard, not extend, a partial argument buffer.)

Suggested fix

  • Apply appendToolCallArgumentDelta on the client-facing relay path, not just the translator paths (or normalise upstream snapshot-style deltas into incremental deltas before relaying).
  • Reset the per-attempt arguments buffer whenever an attempt is abandoned.
  • Consider a JSON.parse assertion on the assembled arguments before finish_reason: "tool_calls" is emitted, as a guard against both cases.

Note on appendToolCallArgumentDelta itself

The existing next.startsWith(existing) rule is the right conservative choice — its comment correctly warns that fuzzy overlap dedup silently truncates legitimate incremental deltas. The gap here is coverage (where it is applied), not the rule. Key-order-changed snapshots are inherently ambiguous from the fragment alone, so a per-attempt reset is likely the safer lever than widening the string heuristic.

Test Impact

Needs a new unit test

Error Logs / Output

undefined

Screenshots

No response

Additional Context

No response

Validation Plan

No response

Activity

0

No activity yet.