#833

Add a PR change risk indicator to the PR panel

Opennicoalbaneseopened 6mo ago

Summary

Add a change risk signal to the PR panel that classifies the current PR as Low, Medium, or High risk.

This is for design/spec discussion only for now — no implementation yet.

Desired behavior

  • Only analyze risk after a PR exists
  • Recompute when the PR head SHA changes
  • Reuse the cached result when the PR head SHA is unchanged
  • Show the result in the open PR panel
  • Initial output should be:
    • risk level: Low | Medium | High
    • short reason

Risk rubric

Risk should weight critical surface area more heavily than raw diff size.

Examples:

  • High: auth, permissions, secrets, billing, data writes/deletes, migrations, infra/config, security-sensitive code
  • Medium: data fetching, business logic, state management, routing, non-trivial backend behavior
  • Low: copy, styling, isolated presentational UI, tests-only, small local refactors

Recommended v1 approach

Use a hybrid approach instead of sending the full raw diff by default:

  1. Build a PR file manifest with:
    • path
    • status
    • additions/deletions
    • file type
  2. Derive local tags such as:
    • auth
    • api
    • migration
    • infra
    • ui
    • tests
  3. Use heuristics first for a baseline risk score
  4. Optionally send the manifest and small sanitized snippets for sensitive files to GPT-5.4 for the final classification

Security / privacy constraints

  • Treat diff contents as untrusted input
  • Use structured output only
  • Redact likely secrets before any model call
  • Skip binaries and generated files
  • Cap input size
  • Cache by PR head SHA to avoid repeated calls
  • Persist only the final risk result, not raw diff content used for prompting

Open questions

  • Should v1 use only file metadata/tags, or include small sanitized snippets for sensitive files?
  • Should the cached result live on the session record keyed by PR head SHA?
  • Should the UI eventually include impacted areas in addition to the short reason?

Acceptance criteria

  • Analysis timing is defined clearly
  • Low/medium/high rubric is documented
  • Model input is minimized and sanitized
  • Cache invalidation is based on PR head SHA
  • No implementation is started as part of this issue

Activity

0

No activity yet.