#1667

Reported usage overcounts OpenAI's actual billed tokens (~1.33x); cache-write cost never priced

Openbhaweshkrsingh-inopened 1d ago

Summary

Two independent, verified deficiencies in LLMUsageLedger's reported run usage, found while cross-checking a Strix run against OpenAI's own /v1/organization/usage/completions API (ground truth, not an assumption).

1. Reported token counts overcount what OpenAI's own Usage API shows for the same window (~1.33x)

Ran Strix (main branch, commit 06015506) with STRIX_LLM_INPUT_COST_PER_MTOK=0.10/OUTPUT=0.50/CACHED_INPUT=0.01 set (real gpt-6-luna rates). Pulled OpenAI's own usage API (1-minute buckets) for the exact run window as independent ground truth:

Strix self-reported (run.json's llm_usage)OpenAI's Usage API, same window
Input tokens152,579,583114,275,763
Cached tokens148,895,044111,384,994
Output tokens688,056501,059
Requests1,217864

Ratio ≈ 1.33x across input/cached/output. The run hit many retries (rate limits + a content-safety flag) during this window.

A second, independent data point from the same run: one agent's line in run.json (input_tokens=6,169,763, cached=5,999,971, output=29,431, cost=$1.2316889) reproduces exactly as correct_cost + cached_tokens × input_rate ($0.6316918 + $0.5999971 = $1.2316889) — i.e. cached tokens billed once correctly and once again at the full rate.

2. Cache-write tokens are never priced at all

_configured_usage_cost() in strix/report/usage.py reads input_tokens and input_tokens_details.cached_tokens, but never input_tokens_details.cache_write_tokens — even though the OpenAI Responses API returns it and OpenAI bills it as a separate, additive line item (confirmed via /v1/organization/costs?group_by=line_item: <model>, cache writes is real and nontrivial, distinct from cached input/input/output).

Suggested fix

  1. For (1): check whether a retried/cancelled request's usage gets recorded via record()/record_provider() separately from its successful retry, or whether an in-flight request cancelled by --max-budget still gets its (possibly never-billed) partial usage counted. A regression test replaying a request-with-one-retry (and a request cancelled mid-flight) should assert totals match only what OpenAI actually bills.
  2. For (2): add a cached_input_write_cost_per_mtok setting (mirroring cached_input_cost_per_mtok) and include cache_write_tokens * write_rate in _configured_usage_cost().

We made both fixes in our own project's cost tracker (a downstream tool using the same configurable-rate pattern) and verified the token-count comparison and cache-write gap independently against real OpenAI billing. Happy to share the run.json data used above if useful.

Activity

0

No activity yet.