#1659

[BUG] Bare Grok pricing test fails after LiteLLM catalog changes

Opensamirhvbropened 3d ago

Describe the bug

tests/test_pricing.py::test_resolves_common_bare_model_names still asserts that the bare name grok-4.5 resolves against LiteLLM's mutable catalog. In the current GitHub Actions environment it returns None, so the test fails even though the resolver is correctly declining to invent a catalog entry or choose between different prices.

To Reproduce

On upstream 09536cf1481731c790995bd8060db19a3b75ff48, use the frozen lockfile and the normal catalog behavior (do not force the bundled fallback map):

uv run --frozen pytest -q tests/test_pricing.py::test_resolves_common_bare_model_names

Current failure:

grok = resolve_litellm_model("grok-4.5")
assert grok is not None
E assert None is not None

The isolated Linux matrix reproduced this on Python 3.12, 3.13 and 3.14: each run finishes with 1 failed, 2532 passed, 3 xfailed after combining #1654, #1657, #1658 and existing #1577. This is the only remaining test failure in that combined tree. With the bundled map on macOS the test passes, confirming its dependence on registry data.

Expected behavior

The bare-name/price-consensus contract should be checked with a synthetic equal-price catalog. Catalog refreshes or removed model aliases should not make a unit test fail without a code change. The provider-qualified examples and differently-priced-provider guard should remain covered.

System Information:

  • OS: Ubuntu GitHub Actions runners
  • Strix baseline: 09536cf1481731c790995bd8060db19a3b75ff48
  • Combined diagnostic tree: 9c7a5d567715628a469d46e2b9addfe3b4c1c801
  • Python: 3.12, 3.13, 3.14
  • LLM: none; catalog lookup only

Additional context

#1213 and closed-unmerged #1215 were reviewed before reporting this. #1215 was closed because de73011 removed the old bare MiniMax assertion. That decision addressed that earlier case; the current failure concerns the remaining bare Grok assertion. This report does not reopen the old proposed patch unchanged.

The fork already has a small local test correction at samirhvbr/strix@9e1af83447392f389771160d2de8040c60cb89c1. I will adapt that existing test-only work to current upstream and ensure the synthetic catalog and resolver cache are restored after the test. No production resolution policy needs to change.

Activity

0

No activity yet.