#2921
Pull Requests/phaseo/Phaseo/#2921

chore(api): load the provider simulator catalogue from the database

OpenDanielButler1wants to merge intofrom
3 commits110 files changed+38393-26980

https://entire.com/gh/phaseoteam/-/trails/153/changes/Phaseo/153

What changed

  • tools/provider-simulator/src/catalogue.ts now reads the catalogue from the database: v2_model_provider_routes, v2_route_capabilities, and active v2_pricing_skus with their billable v2_pricing_sku_meters. It uses SELECTs only, reads paged (1,000 rows per request), and loads once per process through loadCatalogue(). The sync accessors (loadCatalogueRoutes, loadCataloguePriceCard, catalogueRouteRows, catalogueSkus) serve the cached rows.
  • Credentials: SUPABASE_URL/NEXT_PUBLIC_SUPABASE_URL plus SUPABASE_SERVICE_ROLE_KEY, read from the environment, apps/api/.dev.vars, apps/api/.env.local or apps/web/.env.local.
    • Without a service-role key the simulator falls back to the public anon key and prints a warning. Row-level security then hides internal and stealth routes and SKUs not in effect now.
    • With no credentials at all, every *.sim.ts run and simulate:providers:expectations fail with a message saying what to set.
  • The rule shaping in pipeline/pricing/loader.ts moved unchanged into pipeline/pricing/sku-card.ts (buildPriceCardFromSkuRows). loadPriceCard calls it, and so does the simulator, so both shape price cards with the same code.
  • pricing/catalogue-cards.ts builds the request-context-RPC-shaped sweep cards from the same rows. Rule ids are now <sku_code>@v<version>#<meter_key>, prefixed with the route id when a card spans several routes. Each card in expectations/pricing/*.json lists its routes and skus instead of a file path.
  • e2e/context.ts, scripts/build-expectations.ts, the sim entry points, the README, the audits and the e2e nontext.json expectation no longer point at the deleted JSON.
  • The pricing sweep clock defaults to the start of the run (SIM_PRICING_NOW still overrides it).
  • The "unsupported card field" sanity check is gone: SKU rows have no card-level fields that the engine ignores.
  • Expectations (expectations/pricing/*) and results (results/**) are regenerated from the database. expectations/providers/* regenerated with no diff.

Why

#3334 removed the committed catalogue JSON, so the simulator could no longer load routes or prices. The database is the only source of catalogue rows now, and no catalogue snapshot is committed here. The expectation files are test expectations, keyed by database identity.

How

  • Async loading: each sim file awaits loadCatalogue() before it touches the catalogue, before the e2e network guard goes in.
    • Supabase gets the real fetch, captured at import, so the simulator's fetch capture never sees the catalogue reads.
  • Why a separate module: sku-card.ts has no loader or runtime imports. The e2e sim mocks @pipeline/pricing/loader with a factory that imports the harness, which imports the catalogue. Importing the loader from the catalogue deadlocked the e2e sim at import time, so the helper lives in its own module.
  • CI: simulate:providers:test keeps serving its small route and price-card fixture through vi.mock, so it needs no credentials.

Before and after

Before (JSON, committed 10-19 run)After (database, this run)
Catalogue sourcepackages/data/catalog (deleted in #3334)v2 tables, read once per run
Price identity in expectationsfile path + #<index>route ids + <sku_code>@v<version>#<meter_key>
Active routes priced by the sweep1,5711,345
Active routes with a rule in effect1,532 (97.5%)1,327 (98.7%)
Rules exercised6,130 / 6,3816,290 / 6,508
Gate: mismatches / allowed / failing9,033 / 9,033 / 08,548 / 7,887 / 661
Runs without credentialsn/aunit tests yes; sims fail with a clear message

Verification

  • git grep packages/data/catalog -- apps/api/tools/provider-simulator: no matches.
  • pnpm --filter @phaseo/gateway-api typecheck passes. The simulator files also typecheck with a temporary tsconfig that adds tools/provider-simulator: 0 errors.
  • pnpm --filter @phaseo/gateway-api simulate:providers:test (no credentials): 5 files, 52 tests passed.
  • vitest run src/pipeline/pricing: 26 files, 229 tests passed. This covers loader.test.ts after the move.
  • simulate:providers:expectations: 80 provider files, unchanged.
  • I ran the full simulate:providers once.
    • It read the catalogue with the anon key, because no service-role key was available locally: 4,339 routes (1,317 active), 2,472 active SKUs. Every live SKU on an active route is public, so the sweep sees all live prices. Future-dated and expired SKUs and internal routes are not visible.
    • Pricing and e2e passed (75 s). The executor run took 9.6 min, and the gate fails: 661 mismatches (156 groups) are not on the allow-list.
    • Not all of these come from the catalogue. The committed results date from 10-19 (#2968), and the gateway has changed since.

Gate failures by cause (156 groups)

GroupsCauseKind of change
2 (136 mismatches)Chat Completions providers now reject input_image file parts and document_url with unsupported_parameter (#2993, 10-19)gateway change
14e2e error bodies: error is now an object, and 400s keep their status (errors.*.400, failover.all-*)gateway change
4input_audio / file_id parts no longer reach Anthropic, Azure, AI Studio and Mistralgateway change (likely)
44e2e cases pin routes that are no longer live: Vertex google/gemini-2.5-flash (retired 10-22) and OpenAI openai/gpt-image-1 (ended 10-23); the context lookup fails with a 500catalogue change
56Amazon Bedrock's sampled OpenAI route moved from openai/gpt-6-astra to openai/gpt-5.4, and the simulated response loses text, tool calls and usage on every surfacecatalogue change exposing an executor or simulator gap
6Anthropic US and Azure now sample anthropic/claude-opus-5.5, which returns 400 for tool_choice and disabled reasoningcatalogue change
28Haiku 5.5 and the Venice cards now condition on long_context_input_tokens. The engine derives that value from usage (engine.ts:500); the oracle in pricing/calculator.ts doesn't, so it prices $0simulator oracle gap
2openai-eu batch: the cached-write meter is missing on the documented casecatalogue change

Other notable catalogue changes since 09-28

  • Active text routes went to zero for modelscope (38 before), ai21 and sakana, and down to 2 for parasail (35 before). Their pricing expectation files were removed or shrunk.
  • New pricing expectations exist for doubleword, empiriolabs, inference-net, reflection, typesafe and wafer-zdr.
  • Sora 2 has two overlapping active SKUs for 1280x720: $0.10/s (offer-e48eb6ccb12e03192f71, from 2025-09-30) and $0.05/s (offer-1f77c24dd43e9d79056f, from 2026-03-12). The e2e reservation expectation still says $0.10 and passed in this run. The overlap is worth a data check.

Follow-ups, not in this PR

  • Point the e2e cases at live routes.
  • Teach the pricing oracle long_context_input_tokens.
  • Re-run with a service-role key, then triage the failing groups into fixes or allow-list entries.

🤖 Generated with Claude Code

Activity

12
D
chore(api): load the provider simulator catalogue from the database
d0e2b6a
D
chore(api): regenerate provider simulator expectations and results from the database
e56a3e4
vercel[bot]1d ago

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
ProjectDeploymentActionsUpdated
phaseo SkippedOct 29, 2026 4:09pm UTC

coderabbitai[bot]1d ago

IMPORTANT

Review skipped

Too many files!

This PR contains 110 files, which is 10 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

Check out review usage here.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6b4e3410-3623-4725-a0e9-067e3c4d28e2
Commits

Reviewing files that changed from the base of the PR and between 401dbdf6b9b4cacad6be56f9adcb305a9f45081d and 77d272593467a77e912e5e68975e79e9154334a4.

Files selected for processing: 110
  • apps/api/src/pipeline/pricing/loader.ts
  • apps/api/src/pipeline/pricing/sku-card.ts
  • apps/api/tools/provider-simulator/AUDIT-traced-nontext.md
  • apps/api/tools/provider-simulator/AUDIT.md
  • apps/api/tools/provider-simulator/README.md
  • apps/api/tools/provider-simulator/debug.sim.ts
  • apps/api/tools/provider-simulator/e2e/context.ts
  • apps/api/tools/provider-simulator/e2e/e2e.sim.ts
  • apps/api/tools/provider-simulator/e2e/state.ts
  • apps/api/tools/provider-simulator/expectations/e2e/nontext.json
  • apps/api/tools/provider-simulator/expectations/pricing/ai21.json
  • apps/api/tools/provider-simulator/expectations/pricing/aion-labs.json
  • apps/api/tools/provider-simulator/expectations/pricing/akashml.json
  • apps/api/tools/provider-simulator/expectations/pricing/alibaba-cloud.json
  • apps/api/tools/provider-simulator/expectations/pricing/amazon-bedrock.json
  • apps/api/tools/provider-simulator/expectations/pricing/ambient.json
  • apps/api/tools/provider-simulator/expectations/pricing/anthropic-aws-us.json
  • apps/api/tools/provider-simulator/expectations/pricing/anthropic-aws.json
  • apps/api/tools/provider-simulator/expectations/pricing/anthropic-us.json
  • apps/api/tools/provider-simulator/expectations/pricing/anthropic.json
  • apps/api/tools/provider-simulator/expectations/pricing/arcee-ai.json
  • apps/api/tools/provider-simulator/expectations/pricing/atlascloud.json
  • apps/api/tools/provider-simulator/expectations/pricing/avian.json
  • apps/api/tools/provider-simulator/expectations/pricing/azure.json
  • apps/api/tools/provider-simulator/expectations/pricing/baidu.json
  • apps/api/tools/provider-simulator/expectations/pricing/baseten.json
  • apps/api/tools/provider-simulator/expectations/pricing/black-forest-labs.json
  • apps/api/tools/provider-simulator/expectations/pricing/byteplus.json
  • apps/api/tools/provider-simulator/expectations/pricing/canopy-wave.json
  • apps/api/tools/provider-simulator/expectations/pricing/cerebras.json
  • apps/api/tools/provider-simulator/expectations/pricing/cloudflare.json
  • apps/api/tools/provider-simulator/expectations/pricing/cohere.json
  • apps/api/tools/provider-simulator/expectations/pricing/deepinfra.json
  • apps/api/tools/provider-simulator/expectations/pricing/deepseek.json
  • apps/api/tools/provider-simulator/expectations/pricing/doubleword.json
  • apps/api/tools/provider-simulator/expectations/pricing/elevenlabs.json
  • apps/api/tools/provider-simulator/expectations/pricing/empiriolabs.json
  • apps/api/tools/provider-simulator/expectations/pricing/fireworks.json
  • apps/api/tools/provider-simulator/expectations/pricing/friendli.json
  • apps/api/tools/provider-simulator/expectations/pricing/gmicloud.json
  • apps/api/tools/provider-simulator/expectations/pricing/google-ai-studio.json
  • apps/api/tools/provider-simulator/expectations/pricing/google-vertex-eu.json
  • apps/api/tools/provider-simulator/expectations/pricing/google-vertex.json
  • apps/api/tools/provider-simulator/expectations/pricing/groq.json
  • apps/api/tools/provider-simulator/expectations/pricing/inception.json
  • apps/api/tools/provider-simulator/expectations/pricing/inference-net.json
  • apps/api/tools/provider-simulator/expectations/pricing/inflection.json
  • apps/api/tools/provider-simulator/expectations/pricing/io-net.json
  • apps/api/tools/provider-simulator/expectations/pricing/ionrouter.json
  • apps/api/tools/provider-simulator/expectations/pricing/longcat.json
  • apps/api/tools/provider-simulator/expectations/pricing/ltx.json
  • apps/api/tools/provider-simulator/expectations/pricing/meta.json
  • apps/api/tools/provider-simulator/expectations/pricing/minimax-lightning.json
  • apps/api/tools/provider-simulator/expectations/pricing/minimax.json
  • apps/api/tools/provider-simulator/expectations/pricing/mistral-eu.json
  • apps/api/tools/provider-simulator/expectations/pricing/mistral.json
  • apps/api/tools/provider-simulator/expectations/pricing/modelscope.json
  • apps/api/tools/provider-simulator/expectations/pricing/moonshotai.json
  • apps/api/tools/provider-simulator/expectations/pricing/morph.json
  • apps/api/tools/provider-simulator/expectations/pricing/morpheus.json
  • apps/api/tools/provider-simulator/expectations/pricing/nebius-token-factory.json
  • apps/api/tools/provider-simulator/expectations/pricing/novita.json
  • apps/api/tools/provider-simulator/expectations/pricing/openai-eu.json
  • apps/api/tools/provider-simulator/expectations/pricing/openai.json
  • apps/api/tools/provider-simulator/expectations/pricing/ovhcloud.json
  • apps/api/tools/provider-simulator/expectations/pricing/parasail.json
  • apps/api/tools/provider-simulator/expectations/pricing/perplexity.json
  • apps/api/tools/provider-simulator/expectations/pricing/poolside.json
  • apps/api/tools/provider-simulator/expectations/pricing/reflection.json
  • apps/api/tools/provider-simulator/expectations/pricing/reka.json
  • apps/api/tools/provider-simulator/expectations/pricing/relace.json
  • apps/api/tools/provider-simulator/expectations/pricing/sail-research.json
  • apps/api/tools/provider-simulator/expectations/pricing/sakana.json
  • apps/api/tools/provider-simulator/expectations/pricing/sambanova.json
  • apps/api/tools/provider-simulator/expectations/pricing/siliconflow.json
  • apps/api/tools/provider-simulator/expectations/pricing/spacex-ai.json
  • apps/api/tools/provider-simulator/expectations/pricing/stepfun.json
  • apps/api/tools/provider-simulator/expectations/pricing/streamlake.json
  • apps/api/tools/provider-simulator/expectations/pricing/tensorix.json
  • apps/api/tools/provider-simulator/expectations/pricing/together.json
  • apps/api/tools/provider-simulator/expectations/pricing/typesafe.json
  • apps/api/tools/provider-simulator/expectations/pricing/upstage.json
  • apps/api/tools/provider-simulator/expectations/pricing/venice.json
  • apps/api/tools/provider-simulator/expectations/pricing/voyage.json
  • apps/api/tools/provider-simulator/expectations/pricing/wafer-zdr.json
  • apps/api/tools/provider-simulator/expectations/pricing/wafer.json
  • apps/api/tools/provider-simulator/expectations/pricing/weights-and-biases.json
  • apps/api/tools/provider-simulator/expectations/pricing/xiaomi.json
  • apps/api/tools/provider-simulator/expectations/pricing/z-ai.json
  • apps/api/tools/provider-simulator/pricing/cases.ts
  • apps/api/tools/provider-simulator/pricing/catalogue-cards.ts
  • apps/api/tools/provider-simulator/pricing/pricing.sim.ts
  • apps/api/tools/provider-simulator/pricing/sanity.ts
  • apps/api/tools/provider-simulator/pricing/summary.ts
  • apps/api/tools/provider-simulator/results/COVERAGE.md
  • apps/api/tools/provider-simulator/results/EXPECTATIONS.md
  • apps/api/tools/provider-simulator/results/MATRIX.md
  • apps/api/tools/provider-simulator/results/e2e/SUMMARY.md
  • apps/api/tools/provider-simulator/results/e2e/coverage.json
  • apps/api/tools/provider-simulator/results/e2e/mismatches.json
  • apps/api/tools/provider-simulator/results/pricing/SUMMARY.md
  • apps/api/tools/provider-simulator/results/pricing/coverage.json
  • apps/api/tools/provider-simulator/results/pricing/mismatches.json
  • apps/api/tools/provider-simulator/results/summary.json
  • apps/api/tools/provider-simulator/scripts/build-expectations.ts
  • apps/api/tools/provider-simulator/scripts/refresh-specs.ts
  • apps/api/tools/provider-simulator/simulate.sim.ts
  • apps/api/tools/provider-simulator/src/catalogue.ts
  • apps/api/tools/provider-simulator/src/providers.ts
  • apps/api/tools/provider-simulator/src/report.ts

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

chatgpt-codex-connector[bot]1d ago

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

ReviewStatusCommitReview trigger
📝 Code Review✅ Completed 2026-10-29T15:11:10.628074Ze56a3e4PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions[bot]1d ago

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None
chatgpt-codex-connector[bot]reviewed1d ago
chatgpt-codex-connector[bot]reviewed1d ago
chatgpt-codex-connector[bot]1d ago

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e56a3e4bb0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

D
fix(api): skip capabilities outside their effective window in the simulator
77d2725
DanielButler1reviewed1d ago
DanielButler1reviewed1d ago