#2918

feat(gateway): department budgets now refuse API-key requests once used up

OpenDanielButler1wants to merge intofrom

https://entire.com/gh/phaseoteam/-/trails/150/changes/Phaseo/150

What changed

  • GATEWAY_SPEND_LIMIT_DO_MODE goes from "shadow" to "enforce" in every gateway config. The spend-limit coordinator now decides key caps, department budgets and workspace budgets for API-key requests. Once a department has used its daily, weekly or monthly budget, API-key requests that count toward it return 429 key_limit_exceeded with reason department_<interval>_cost_budget_reached, department_id, department_name, reset_at and Retry-After.
  • The coordinator wait is capped at 2 s (ENFORCE_WAIT_MS). If the object or its Postgres seed stalls, the legacy check decides instead of the request hanging, and a late admission is released.
  • Copy now says department budgets block instead of "alerts only":
    • Settings → Budgets: badge "Blocks at 100%", description, limit help and delete dialog, in 9 locales.
    • limits.mdx and its 8 translations.
    • OpenAPI create/delete descriptions, localized copy and the generated TS SDK comments.
    • Terraform provider docs and schema description.
    • The 100% push body, via migration 20261029134622_department_budgets_block_requests.sql: "Requests from the department's API keys are now refused until the budget resets."
    • The table comment.
  • The docs state the exceptions:
    • batch jobs, realtime sessions and OAuth requests are not blocked;
    • if the budget check is briefly unavailable, requests are allowed;
    • alerts come from usage that can lag by up to an hour.

Why

This is step 3 of 3 for the user's request to make department budgets block ("sort the not finished parts", then "full send this"). #3347 added department budgets to the coordinator. #3358 turned on shadow mode. The live shadow test on production was posted on #3358: the coordinator denied with department_daily_cost_budget_reached at 38,500 / 20,000 nanos while the legacy check allowed, and that figure matched gateway_requests exactly.

How

  • Config flip. The enforce wiring was reviewed and merged in #3347.
  • Wait cap. It came from review: without it, a stalled coordinator call would have stalled ordinary API-key traffic.
  • Database change. It is hand-written because there is no Docker here. The migration equals the declarative files; a PGlite test asserts that and the new push body.

Before and after

BeforeAfter
API-key request once its department budget is used up200, push alert only429 key_limit_exceeded, reason department_daily_cost_budget_reached, department named, Retry-After until reset
Coordinator stalls for more than 2 sn/a (shadow, background only)Legacy check decides; request continues
Settings badge"Alerts only""Blocks at 100%"
100% push body"Requests keep running: department budgets send alerts but don't block.""Requests from the department's API keys are now refused until the budget resets."

Verification

  • pnpm exec vitest run src/core/spend-limits.test.ts src/pipeline/before/guards.spend-limit.test.ts src/pipeline/after/audit.spend-limit-shadow.test.ts → 38 passed.
  • The new stall test fails without the wait cap ("Test timed out in 60000ms") and passes with it.
  • Wrangler-reading tests (regional-broadcast, webhook-egress-config) and spend-limit-alerts/department-budgets route tests → passed.
  • tsc --noEmit (apps/api) → clean.
  • PGlite: supabase/tests/department-spend-budgets.test.mjs → ok; gateway-spend-limit-seed.test.mjs → passed.
  • Docs: generate-localized-openapi.mjs --check, validate-locales.mjs (654 pages), translation-freshness.mjs check, pnpm run docs:public-safety (6,252 files) → all pass.
  • Web: validate-message-scopes.mjs --check-client-scopes and --self-test, validate-localisation.ts, generate-docs-search.mjs --check → pass.
  • Terraform: gofmt -l clean, go build ./... ok.
  • Not run locally: db:schema:check (needs Docker), Mintlify docs:build, web Jest.
  • Live proof after deploy: the same temporary department, now over its budget, should get a 429 naming it. Results will be posted on this PR.

Proved with unit tests (including one that fails without the wait cap), PGlite migration tests and docs/i18n checks; enforcement proved live after deploy (PR comment).

Reviewed with sol-loop: 2 rounds with gpt-6.1-sol on high. Sol verdict: approved

Model: Claude Opus 5.5 (medium effort) · Harness: Claude Code in T3 Code

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Department budgets now block eligible API-key requests once daily, weekly, or monthly limits are exhausted, returning 429 key_limit_exceeded with reset details. Batch jobs, realtime sessions, and OAuth requests are excluded.
    • Owners and admins receive alerts at approximately 80% and 100% of budget usage. Usage reporting may lag by up to an hour.
  • Bug Fixes
    • If budget enforcement is temporarily unavailable, requests are allowed. Enforcement checks now time out after two seconds to avoid leaving requests waiting.
  • Documentation
    • Updated department-budget guidance and translations to explain enforcement, exceptions, alerts, and reset behavior.

Activity

55
vercel[bot]1d ago

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated
phaseo ReadyPreviewOct 29, 2026 2:48pm UTC

chatgpt-codex-connector[bot]1d ago

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

ReviewStatusCommitReview trigger
📝 Code Review✅ Completed 2026-10-29T14:13:50.197681Z6b815f0PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions[bot]1d ago

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None
coderabbitai[bot]1d ago

WARNING

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: bbb9a476-3b8e-48f2-9f8b-71bdc908cee3
Commits

Reviewing files that changed from the base of the PR and between 6b815f0cc48198b6317bd77e93fd16b4eacfbdd2 and fed35f36fe8e51002742380e67bc4a32aa93f2dc.

Files selected for processing: 27
  • apps/docs/ar/openapi/v1/openapi.localized.yaml
  • apps/docs/de/openapi/v1/openapi.localized.yaml
  • apps/docs/es/openapi/v1/openapi.localized.yaml
  • apps/docs/fr/openapi/v1/openapi.localized.yaml
  • apps/docs/hi/openapi/v1/openapi.localized.yaml
  • apps/docs/ja/openapi/v1/openapi.localized.yaml
  • apps/docs/openapi/localized-copy.json
  • apps/docs/openapi/v1/openapi.yaml
  • apps/docs/pt-BR/openapi/v1/openapi.localized.yaml
  • apps/docs/pt-BR/v1/api-reference/limits.mdx
  • apps/docs/translation-status.json
  • apps/docs/zh-Hans/openapi/v1/openapi.localized.yaml
  • apps/web/messages/ar-SA/settings-ui.json
  • apps/web/messages/de-DE/settings-ui.json
  • apps/web/messages/en-GB/settings-ui.json
  • apps/web/messages/es-ES/settings-ui.json
  • apps/web/messages/fr-FR/settings-ui.json
  • apps/web/messages/hi/settings-ui.json
  • apps/web/messages/ja/settings-ui.json
  • apps/web/messages/pt-BR/settings-ui.json
  • apps/web/messages/zh-Hans/settings-ui.json
  • packages/integrations/terraform-provider-phaseo/docs/resources/department_budget.md
  • packages/integrations/terraform-provider-phaseo/internal/provider/resource_department_budget.go
  • packages/sdk/sdk-ts/src/oapi-gen/client/default.ts
  • supabase/migrations/20261029134622_department_budgets_block_requests.sql
  • supabase/schemas/public/functions/sweep_department_budget_alerts.sql
  • supabase/tests/department-spend-budgets.test.mjs
Walkthrough

The change switches gateway spend-limit coordination from shadow to enforce, bounds coordinator waits, and uses the legacy key-limit check when the coordinator is unavailable. It updates department-budget alerts and revises dashboard, API, SDK, Terraform, and localized documentation for request blocking and its exceptions.

Changes: Department Budget Enforcement

Layer / File(s)Summary
Bound coordinator decisions
apps/api/src/core/spend-limits.ts, apps/api/src/core/spend-limits.test.ts
Coordinator decisions now have a 2,000 ms wait. A timeout returns spend_limit_timeout; tests verify the timeout, late admission release, and the following request.
Enable enforcement and update alerts
apps/api/wrangler*.toml, supabase/migrations/*department_budgets_block_requests.sql, supabase/schemas/public/functions/sweep_department_budget_alerts.sql, supabase/schemas/public/tables/workspace_department_budgets.sql, supabase/tests/department-spend-budgets.test.mjs, apps/api/src/pipeline/notifications/spend-limit-alerts.ts, apps/api/src/routes/v1/control/department-budgets.ts, apps/web/src/app/.../settings/budgets/actions.ts, .changeset/department-budgets-block.md
Worker configurations switch coordinator mode from shadow to enforce. The migration defines the department-budget alert sweep and notification content; tests check the migration and alert text. Supporting comments describe enforcement, reseeding, and alert timing.
Update budget settings copy
apps/web/src/components/(gateway)/settings/budgets/DepartmentBudgetsSection.tsx, apps/web/src/lib/fetchers/internal/settingsTypes.ts, apps/web/messages/*/settings-ui.json
Settings descriptions and translations state that eligible API-key requests are rejected when budgets are exhausted, identify excluded request types, retain alert thresholds, and describe deletion effects.
Update API and integration documentation
apps/docs/openapi/*, apps/docs/*/openapi/v1/openapi.localized.yaml, apps/docs/v1/api-reference/limits.mdx, apps/docs/*/v1/api-reference/limits.mdx, apps/docs/openapi/localized-copy.json, apps/docs/translation-status.json, packages/sdk/sdk-ts/src/oapi-gen/client/default.ts, packages/integrations/terraform-provider-phaseo/{docs/resources/department_budget.md,internal/provider/resource_department_budget.go}
OpenAPI descriptions, API references, SDK documentation, and Terraform descriptions now cover rejection responses, exclusions, reset behavior, alerts, and budget deletion. Translation checksums are updated for the revised localized documentation.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Request as API-key request
  participant Coordinator as Spend-limit coordinator
  participant Legacy as Legacy key-limit check
  Request->>Coordinator: Request spend-limit decision
  Coordinator-->>Request: Enforce decision or unavailable timeout
  Request->>Legacy: Check limit if coordinator is unavailable
  Legacy-->>Request: Fallback decision

Possibly related PRs

  • phaseoteam/Phaseo#3208: Introduces department-budget management and the settings UI whose enforcement behavior this change updates.
  • phaseoteam/Phaseo#3343: Adds an integration key with a monthly spend limit governed by API-key spend-limit enforcement.
  • phaseoteam/Phaseo#3243: Adds workload-identity authentication that uses API-key-shaped results and references the shared limits and budgets behavior.
Merge Risk: Moderate · previous review 6b815

Enabling department budget blocking has edge cases when the budget check times out: spend can briefly exceed a budget, or a budget can reject requests slightly early. The API contract also understates how long Retry-After can be. Resolve the lease and double-count handling before relying on budgets as a strict cap; the other findings are documentation and copy fixes.

Pre-merge checks: 5
Check nameStatusExplanation
Description CheckCheck skipped - CodeRabbit’s high-level summary is enabled.
Title checkThe title clearly and concisely describes the main change: department budgets now reject API-key requests after the budget is exhausted.
Docstring CoverageDocstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues checkCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes checkCheck skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 9 files. (40 skipped: 39 unsupported, 1 too large.)

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
coderabbitai[bot]reviewed1d ago
DanielButler11d ago

Live enforce test after deploy (2026-10-29 16:18 UTC, owner's Personal workspace, temporary key linked to a temporary department with a 20,000-nano daily budget that was already used up):

429  Retry-After: 27695   (midnight UTC)
code: key_limit_exceeded
message: "The Budget block test (temporary) department has reached its daily budget ($0.000173 of $0.00002)."

Before the deploy, the same key got 200s, because shadow mode only logs. The public body dropped reason, reset_at, department_id and department_name, which handleError filters out for every key_limit_exceeded. That is fixed in #3386.