Summary
Two independent, verified deficiencies in LLMUsageLedger's reported run usage, found while cross-checking a Strix run against OpenAI's own /v1/organization/usage/completions API (ground truth, not an assumption).
1. Reported token counts overcount what OpenAI's own Usage API shows for the same window (~1.33x)
Ran Strix (main branch, commit 06015506) with STRIX_LLM_INPUT_COST_PER_MTOK=0.10/OUTPUT=0.50/CACHED_INPUT=0.01 set (real gpt-6-luna rates). Pulled OpenAI's own usage API (1-minute buckets) for the exact run window as independent ground truth:
Strix self-reported (run.json's llm_usage) | OpenAI's Usage API, same window | |
|---|---|---|
| Input tokens | 152,579,583 | 114,275,763 |
| Cached tokens | 148,895,044 | 111,384,994 |
| Output tokens | 688,056 | 501,059 |
| Requests | 1,217 | 864 |
Ratio ≈ 1.33x across input/cached/output. The run hit many retries (rate limits + a content-safety flag) during this window.
A second, independent data point from the same run: one agent's line in run.json (input_tokens=6,169,763, cached=5,999,971, output=29,431, cost=$1.2316889) reproduces exactly as correct_cost + cached_tokens × input_rate ($0.6316918 + $0.5999971 = $1.2316889) — i.e. cached tokens billed once correctly and once again at the full rate.
2. Cache-write tokens are never priced at all
_configured_usage_cost() in strix/report/usage.py reads input_tokens and input_tokens_details.cached_tokens, but never input_tokens_details.cache_write_tokens — even though the OpenAI Responses API returns it and OpenAI bills it as a separate, additive line item (confirmed via /v1/organization/costs?group_by=line_item: <model>, cache writes is real and nontrivial, distinct from cached input/input/output).
Suggested fix
- For (1): check whether a retried/cancelled request's
usagegets recorded viarecord()/record_provider()separately from its successful retry, or whether an in-flight request cancelled by--max-budgetstill gets its (possibly never-billed) partial usage counted. A regression test replaying a request-with-one-retry (and a request cancelled mid-flight) should assert totals match only what OpenAI actually bills. - For (2): add a
cached_input_write_cost_per_mtoksetting (mirroringcached_input_cost_per_mtok) and includecache_write_tokens * write_ratein_configured_usage_cost().
We made both fixes in our own project's cost tracker (a downstream tool using the same configurable-rate pattern) and verified the token-count comparison and cache-write gap independently against real OpenAI billing. Happy to share the run.json data used above if useful.
Activity
0No activity yet.