#1622

[BUG] Resuming with another model reprices historical agent usage and misapplies the budget

Opensamirhvbropened 9d ago

Describe the bug When a scan is resumed with a different model and configured token prices, the first new response changes the model attached to the agent's entire accumulated usage. All historical tokens for that agent are then priced at the new model's rate.

This can either stop a scan before its actual budget is reached or reduce its accumulated cost after a paid request, allowing the budget threshold to be missed. The incorrect total is also persisted in run.json and consumed by the UI.

To Reproduce On current main (6b7ae2ebe4d47037a927aade23265ff4e4f27e84), run uv sync --frozen --python 3.12, then execute this from the checkout. It uses a temporary run directory, real ReportState serialization/hydration, and synthetic SDK usage. Only the external model-price estimator is stubbed; no provider request, credential, Docker sandbox, or scan is needed.

test_home=$(mktemp -d)
env -i HOME="$test_home" PATH="$PATH" LITELLM_MODE=PRODUCTION \
  STRIX_TELEMETRY=0 STRIX_NO_UPDATE_CHECK=1 .venv/bin/python - <<'PY'
import json
import os
import tempfile
from unittest.mock import patch

from agents.usage import Usage

from strix.config import loader
from strix.report.state import ReportState


def configure(model, rate):
    os.environ["STRIX_LLM"] = model
    os.environ["STRIX_LLM_INPUT_COST_PER_MTOK"] = str(rate)
    os.environ["STRIX_LLM_OUTPUT_COST_PER_MTOK"] = "0"
    loader._cached = None  # Simulate a fresh process loading the new configuration.


with tempfile.TemporaryDirectory(prefix="strix-resume-pricing-") as directory:
    os.chdir(directory)
    for new_rate, budget in [(8, 0.012), (0.5, 0.0023)]:
        run_name = f"model-switch-{new_rate}"
        usage = Usage(requests=1, input_tokens=1000, total_tokens=1000)
        # Synthetic unknown models: no LiteLLM prices or provider calls are needed.
        with patch("strix.report.usage._estimate_litellm_cost", return_value=None):
            configure("gateway/model-a", 2)
            original = ReportState(run_name)
            original.record_sdk_usage(agent_id="root", usage=usage, model="gateway/model-a")
            initial_cost = original.get_total_llm_cost()

            configure("gateway/model-b", new_rate)
            resumed = ReportState(run_name)
            resumed.hydrate_from_run_dir()
            hydrated_cost = resumed.get_total_llm_cost()
            resumed.record_sdk_usage(agent_id="root", usage=usage, model="gateway/model-b")
            record = json.loads((resumed.get_run_dir() / "run.json").read_text())
            actual = record["llm_usage"]["cost"]
            expected = initial_cost + 1000 * new_rate / 1_000_000
            print(json.dumps({
                "new_input_rate": new_rate,
                "initial_cost": initial_cost,
                "hydrated_cost": hydrated_cost,
                "actual_after_one_new_request": actual,
                "expected_after_one_new_request": expected,
                "budget": budget,
                "actual_budget_reached": actual >= budget,
                "expected_budget_reached": expected >= budget,
            }))
PY

Both examples begin with 1,000 input tokens from model A at $2/Mtok, costing $0.002. Hydration correctly preserves that $0.002 before the new response. Each then records 1,000 additional input tokens for the same agent ID, using model B:

Model B input rateExpected cumulative costActual cost written to run.jsonExample budget effect
$8/Mtok$0.010$0.016A $0.012 threshold is incorrectly reached.
$0.50/Mtok$0.0025$0.001A $0.0023 threshold is missed; total cost falls below the pre-request $0.002.

Expected behavior Usage belonging to model A should not be attributed to model B when the same agent resumes. Adding a response with positive cost should not reduce the run's accumulated cost. Preserve the existing behavior of applying configured prices to usage from the matching model.

System Information:

  • OS: macOS 27.0.1, arm64
  • Strix Version or Commit: 1.7.0 / 6b7ae2ebe4d47037a927aade23265ff4e4f27e84
  • Python Version: 3.12.14, dependencies from uv.lock
  • LLM Used: synthetic gateway/model-a and gateway/model-b; no live LLM used

Additional context LLMUsageLedger.record() accumulates tokens by agent_id and overwrites that agent's model metadata. _configured_agent_cost() subsequently applies the current model's rate to the agent's entire accumulated Usage. Resumed root and child agents retain their IDs, while run_strix_scan() obtains the model from the current settings. ReportUsageHooks.on_llm_end() records the response and immediately checks get_total_llm_cost() against the budget, so this affects execution control as well as display.

Related work reviewed:

  • #1616 introduced configured pricing. Its accepted test_hydrated_usage_uses_configured_rates_not_persisted_cost intentionally reprices persisted usage for the same model; this report concerns tokens from a different model being reassigned after the next response. It does not propose undoing same-model repricing.
  • The dedupe pricing review on #1616 led to limiting configured rates to matching models. That distinction works for separate agent IDs, but an agent that changes model on resume loses the distinction.
  • #1401 concerns duplicate streaming-cost callbacks, a different path. This reproducer uses only SDK usage and manual prices, with no streaming callback.

A possible direction is to retain usage/cost by (agent_id, model) while preserving the aggregate agent view, or to retain a historical-cost component when an agent changes model. The persisted format and compatibility policy need a maintainer decision, especially for old records that already combine multiple models. I have not implemented a schema or repricing-policy change.

Validation is limited to the local persistence/usage path above. No real provider billing, live scan, or rendered UI was exercised.

Activity

0

No activity yet.