#1131

feat: add dataset grain and aggregate reconciliation rules

Openiamgpwants to merge intofrom

Goal and scope

Declare composite row grain and aggregate conservation through Phlo's existing neutral quality API. Second PR in the stack; its base contains explicit event-time freshness.

Change map and risk

  • Add public grain() and reconcile_aggregate() factories, translated by Pandera into bounded SQL checks.
  • Support independent equality filters, existing partition scopes, runtime-selected logical relations, and decimal-exact scalar comparison.
  • Preserve Dataset severity, evidence, and publication policy. Reject pinned catalog references and conflicting evidence names.
  • Combine strict freshness and dataset-constraint handling for empty data while retaining the legacy skip for other checks.
  • Add a disposable real-lakehouse demonstration and author documentation.

Original verification

  • Original head: dataset constraints, public exports, and neutral-provider suites: 59 passed, including mixed explicit-freshness/constraint execution.
  • Identical before/after procedure on Trino 477, Nessie 0.108.3, Iceberg/MinIO and PostgreSQL: a grouped join inflates settled revenue 41 → 52 while grain/range checks pass. Baseline publishes; reconciliation reports delta 11 and blocker policy retains durable draft state. Warning policy publishes without changing the failed quality verdict.
  • Repaired candidate source/target totals 54, while main remains 41, proving both relations follow the candidate ref. This does not prove physical Nessie promotion or cross-table snapshot locking.
  • Reproduce: uv run --locked python examples/lakehouses/wap-failure-lab/scripts/constraints_demo.py --output /tmp/constraints.json. Docker and make setup are required; the script provisions disposable services. See the WAP Failure Lab README for declarations.
  • Full integrated baseline: 7,417 passed; core regression: 447; real storage regression: 101; installed-wheel verification: 40 packages. These broader checks cover the final stack.
  • Links and generated references pass. Final combined production documentation rendering exceeded the development orb's memory limit.
  • New feature: regression coverage includes composite/null keys, scoped aggregates, empty data, tolerance boundaries, evidence-name collisions, and interaction with explicit freshness.

Review corrections

  • Legacy checks remain skipped on empty data even alongside grain or reconciliation checks. Explicit freshness and dataset constraints still execute.
  • Accepted-values checks participate in collision validation using their emitted evidence name, not their display name.
  • Schema and SQL checks select the same runtime Trino ref through the shared resolver. Opposite main/candidate contents reproduce the mismatch before the fix and agree after it. The regression uses real DuckDB SQL behind the Trino driver interface, not a live Trino server.
  • All 269 selected Pandera tests pass. The original head also passed GitHub CI, including documentation rendering.

Post-correction integrated stack: make check passes with 7,419 tests, four skipped and 194 deselected; make test-core-regression passes all 447 tests. These are combined-stack results, not a separate full baseline at this isolated head.

Recovery and maintainer decisions

Opt-in; no migration or deployment. Remove constraint declarations before reverting their implementation. A source revert does not undo publication. No source dependency inference or private-provider compatibility claim. Maintainer review is pending; no independent reviewer or release qualification is claimed.


    

Activity

9
coldtea-pr-lens[bot]4d ago
  • Redraw · drawn at 0222cdb, the branch has moved on

Architecture

Inside the changed components — 1 view

Component view — Pandera Quality Provider

Internal execution components for query-backed dataset constraints in the Pandera provider.

Data flow


Review

2 findings on the diff · 1 addressed · reviewed 0222cdb

TIP

The diagrams are links. Click one to open it on the canvas, then press W or click play to walk through the change

🪧 More tips
  • Run npx skills add coldteadotai/pr-lens, then tell your coding agent: "Diagram the change you just made with PR Lens and attach it to the pull request."
  • Run npx @coldtea/pr-lens-cli analyze --base origin/main on a branch, then npx @coldtea/pr-lens-cli render .pr-lens/graph.json. Same lenses, your own model key, before the pull request exists
  • Click the link under each diagram to open it on a canvas you can zoom, pan and step through
  • Open a diagram on the canvas, then press W or click play to walk through the change one step at a time
  • The CLI's render reads .github/pr-lens.yml and applies your renames, exclusions and lane pins at draw time
  • Set github.comment.collapsed: true in .github/pr-lens.yml to fold the comment behind one View architecture and data flow row. Drawing still runs as before
  • Set github.draw: on-demand in .github/pr-lens.yml and PR Lens stops drawing pull requests on its own. Comment @pr-lens draw on a pull request when you want that one drawn
  • Add .github/workflows/pr-lens.yml with coldteadotai/pr-lens/packages/action@v0 and your model provider's key as its api-key to run PR Lens from your own CI. Any /chat/completions endpoint works
  • After a push, tick Redraw to draw the latest commit
  • Switch GitHub to dark mode and the diagrams follow. The moving dots are this pull request's data in motion
  • Set github.description: false in .github/pr-lens.yml to stop PR Lens adding buttons to your PR descriptions

Thanks for using PR Lens! It's free for OSS. If it helped, a star helps other teams find it. 

coldtea-pr-lens[bot]reviewed4d ago
coldtea-pr-lens[bot]reviewed4d ago
coldtea-pr-lens[bot]reviewed4d ago
coldtea-pr-lens[bot]reviewed4d ago