#1660

test(pricing): isolate bare-name resolution from live catalog changes

Opensamirhvbrwants to merge intofrom

Fixes #1659.

The bare grok-4.5 assertion currently fails on Python 3.12–3.14 in GitHub Actions when LiteLLM's catalog no longer resolves it. A bundled fallback catalog happens to make the same test pass locally. The resolver is correctly following its price-consensus policy; a mutable external example makes the unit test unreliable.

Move the equal-price bare-name contract to a synthetic three-provider catalog and remove the live Grok assertion. Provider-qualified examples and the existing different-price guard remain covered. The temporary catalog is restored by monkeypatch, and the resolver cache is cleared before and after the synthetic case. Production code is unchanged.

This adapts existing fork work from samirhvbr/strix@9e1af83447392f389771160d2de8040c60cb89c1, adding guaranteed cache cleanup and omitting the fork's separate subscription-route guard. I reviewed closed #1215 before preparing this: it was closed because the old MiniMax assertion had been removed by de73011; the remaining Grok assertion is a new occurrence of catalog dependence.

Validation:

  • Before: Linux combined diagnostic, with other pre-existing test failures and the cost-authentication wait corrected, finishes with 1 failed / 2532 passed / 3 xfailed on each of Python 3.12, 3.13 and 3.14; the sole failure is the live Grok assertion.
  • After: pricing and cost tracking modules 43 passed, both with normal catalog loading and with the bundled fallback map (macOS, Python 3.12.14, frozen lockfile, isolated HOME).
  • Repository Ruff lint/format, Mypy and Bandit pass. make check-all stops at the same 941 existing Pyright errors in unchanged production code.
  • git diff --check passes.

The full combined Linux rerun has completed; its result is below. This test-only PR still inherits the unrelated missing-skills fixtures (#1657), missing telemetry settings (#1577), and cost-estimation authentication wait (#1658) until those fixes are present in its base. No real LLM request or scan was used.

Prepared with Codex assistance.

Combined Linux validation

The complete diagnostic matrix at c0651c343e3d7a58391214a2d6b133b14f363bbe combines #1654, #1657, #1658, #1660 and existing #1577 on current upstream 09536cf1. Python 3.12, 3.13 and 3.14 each pass with 2,534 passed / 3 expected xfails, using the frozen lockfile and normal catalog loading. No tests are skipped.

This validates the fixes together. Each independent PR's own upstream run still inherits the other unresolved base defects; the combined diagnostic does not turn those checks green.

Activity

2
greptile-apps[bot]3d ago

Confidence Score: 5/5

[Low impact] This test-only change appears safe to merge.

Summary

Replaces the live grok-4.5 assertion with a synthetic three-provider catalog.

  • The bare-name test checks provider choices from a fixed catalog.

No actionable issues found. Tests were not run during this review.

Reviews (1) · Last reviewed commit: "test(pricing): resolve a bare model name..." · Reviewed by Greptile