Describe the bug When a scan is resumed with a different model and configured token prices, the first new response changes the model attached to the agent's entire accumulated usage. All historical tokens for that agent are then priced at the new model's rate.
This can either stop a scan before its actual budget is reached or reduce its accumulated cost after a paid request, allowing the budget threshold to be missed. The incorrect total is also persisted in run.json and consumed by the UI.
To Reproduce
On current main (6b7ae2ebe4d47037a927aade23265ff4e4f27e84), run uv sync --frozen --python 3.12, then execute this from the checkout. It uses a temporary run directory, real ReportState serialization/hydration, and synthetic SDK usage. Only the external model-price estimator is stubbed; no provider request, credential, Docker sandbox, or scan is needed.
test_home=$(mktemp -d)
env -i HOME="$test_home" PATH="$PATH" LITELLM_MODE=PRODUCTION \
STRIX_TELEMETRY=0 STRIX_NO_UPDATE_CHECK=1 .venv/bin/python - <<'PY'
import json
import os
import tempfile
from unittest.mock import patch
from agents.usage import Usage
from strix.config import loader
from strix.report.state import ReportState
def configure(model, rate):
os.environ["STRIX_LLM"] = model
os.environ["STRIX_LLM_INPUT_COST_PER_MTOK"] = str(rate)
os.environ["STRIX_LLM_OUTPUT_COST_PER_MTOK"] = "0"
loader._cached = None # Simulate a fresh process loading the new configuration.
with tempfile.TemporaryDirectory(prefix="strix-resume-pricing-") as directory:
os.chdir(directory)
for new_rate, budget in [(8, 0.012), (0.5, 0.0023)]:
run_name = f"model-switch-{new_rate}"
usage = Usage(requests=1, input_tokens=1000, total_tokens=1000)
# Synthetic unknown models: no LiteLLM prices or provider calls are needed.
with patch("strix.report.usage._estimate_litellm_cost", return_value=None):
configure("gateway/model-a", 2)
original = ReportState(run_name)
original.record_sdk_usage(agent_id="root", usage=usage, model="gateway/model-a")
initial_cost = original.get_total_llm_cost()
configure("gateway/model-b", new_rate)
resumed = ReportState(run_name)
resumed.hydrate_from_run_dir()
hydrated_cost = resumed.get_total_llm_cost()
resumed.record_sdk_usage(agent_id="root", usage=usage, model="gateway/model-b")
record = json.loads((resumed.get_run_dir() / "run.json").read_text())
actual = record["llm_usage"]["cost"]
expected = initial_cost + 1000 * new_rate / 1_000_000
print(json.dumps({
"new_input_rate": new_rate,
"initial_cost": initial_cost,
"hydrated_cost": hydrated_cost,
"actual_after_one_new_request": actual,
"expected_after_one_new_request": expected,
"budget": budget,
"actual_budget_reached": actual >= budget,
"expected_budget_reached": expected >= budget,
}))
PYBoth examples begin with 1,000 input tokens from model A at $2/Mtok, costing $0.002. Hydration correctly preserves that $0.002 before the new response. Each then records 1,000 additional input tokens for the same agent ID, using model B:
| Model B input rate | Expected cumulative cost | Actual cost written to run.json | Example budget effect |
|---|---|---|---|
| $8/Mtok | $0.010 | $0.016 | A $0.012 threshold is incorrectly reached. |
| $0.50/Mtok | $0.0025 | $0.001 | A $0.0023 threshold is missed; total cost falls below the pre-request $0.002. |
Expected behavior Usage belonging to model A should not be attributed to model B when the same agent resumes. Adding a response with positive cost should not reduce the run's accumulated cost. Preserve the existing behavior of applying configured prices to usage from the matching model.
System Information:
- OS: macOS 27.0.1, arm64
- Strix Version or Commit: 1.7.0 /
6b7ae2ebe4d47037a927aade23265ff4e4f27e84 - Python Version: 3.12.14, dependencies from
uv.lock - LLM Used: synthetic
gateway/model-aandgateway/model-b; no live LLM used
Additional context
LLMUsageLedger.record() accumulates tokens by agent_id and overwrites that agent's model metadata. _configured_agent_cost() subsequently applies the current model's rate to the agent's entire accumulated Usage. Resumed root and child agents retain their IDs, while run_strix_scan() obtains the model from the current settings. ReportUsageHooks.on_llm_end() records the response and immediately checks get_total_llm_cost() against the budget, so this affects execution control as well as display.
Related work reviewed:
- #1616 introduced configured pricing. Its accepted
test_hydrated_usage_uses_configured_rates_not_persisted_costintentionally reprices persisted usage for the same model; this report concerns tokens from a different model being reassigned after the next response. It does not propose undoing same-model repricing. - The dedupe pricing review on #1616 led to limiting configured rates to matching models. That distinction works for separate agent IDs, but an agent that changes model on resume loses the distinction.
- #1401 concerns duplicate streaming-cost callbacks, a different path. This reproducer uses only SDK usage and manual prices, with no streaming callback.
A possible direction is to retain usage/cost by (agent_id, model) while preserving the aggregate agent view, or to retain a historical-cost component when an agent changes model. The persisted format and compatibility policy need a maintainer decision, especially for old records that already combine multiple models. I have not implemented a schema or repricing-policy change.
Validation is limited to the local persistence/usage path above. No real provider billing, live scan, or rendered UI was exercised.
Activity
0No activity yet.