#1131
feat: add dataset grain and aggregate reconciliation rules
Goal and scope
Declare composite row grain and aggregate conservation through Phlo's existing neutral quality API. Second PR in the stack; its base contains explicit event-time freshness.
Change map and risk
- Add public
grain()andreconcile_aggregate()factories, translated by Pandera into bounded SQL checks. - Support independent equality filters, existing partition scopes, runtime-selected logical relations, and decimal-exact scalar comparison.
- Preserve Dataset severity, evidence, and publication policy. Reject pinned catalog references and conflicting evidence names.
- Combine strict freshness and dataset-constraint handling for empty data while retaining the legacy skip for other checks.
- Add a disposable real-lakehouse demonstration and author documentation.
Original verification
- Original head: dataset constraints, public exports, and neutral-provider suites: 59 passed, including mixed explicit-freshness/constraint execution.
- Identical before/after procedure on Trino 477, Nessie 0.108.3, Iceberg/MinIO and PostgreSQL: a grouped join inflates settled revenue 41 → 52 while grain/range checks pass. Baseline publishes; reconciliation reports delta 11 and blocker policy retains durable draft state. Warning policy publishes without changing the failed quality verdict.
- Repaired candidate source/target totals 54, while main remains 41, proving both relations follow the candidate ref. This does not prove physical Nessie promotion or cross-table snapshot locking.
- Reproduce:
uv run --locked python examples/lakehouses/wap-failure-lab/scripts/constraints_demo.py --output /tmp/constraints.json. Docker andmake setupare required; the script provisions disposable services. See the WAP Failure Lab README for declarations. - Full integrated baseline: 7,417 passed; core regression: 447; real storage regression: 101; installed-wheel verification: 40 packages. These broader checks cover the final stack.
- Links and generated references pass. Final combined production documentation rendering exceeded the development orb's memory limit.
- New feature: regression coverage includes composite/null keys, scoped aggregates, empty data, tolerance boundaries, evidence-name collisions, and interaction with explicit freshness.
Review corrections
- Legacy checks remain skipped on empty data even alongside grain or reconciliation checks. Explicit freshness and dataset constraints still execute.
- Accepted-values checks participate in collision validation using their emitted evidence name, not their display name.
- Schema and SQL checks select the same runtime Trino ref through the shared resolver. Opposite main/candidate contents reproduce the mismatch before the fix and agree after it. The regression uses real DuckDB SQL behind the Trino driver interface, not a live Trino server.
- All 269 selected Pandera tests pass. The original head also passed GitHub CI, including documentation rendering.
Post-correction integrated stack: make check passes with 7,419 tests, four skipped and 194 deselected; make test-core-regression passes all 447 tests. These are combined-stack results, not a separate full baseline at this isolated head.
Recovery and maintainer decisions
Opt-in; no migration or deployment. Remove constraint declarations before reverting their implementation. A source revert does not undo publication. No source dependency inference or private-provider compatibility claim. Maintainer review is pending; no independent reviewer or release qualification is claimed.
Activity
9coldtea-pr-lens[bot]reviewed4d ago
coldtea-pr-lens[bot]reviewed4d ago
coldtea-pr-lens[bot]reviewed4d ago
coldtea-pr-lens[bot]reviewed4d ago
0222cdb, the branch has moved onArchitecture
Inside the changed components — 1 view
Component view — Pandera Quality Provider
Internal execution components for query-backed dataset constraints in the Pandera provider.
Data flow
Review
2 findings on the diff · 1 addressed · reviewed
0222cdbisinstance(check, DatasetConstraintCheck)disables the empty-partition skip for every… ·phlo_pandera/decorator.py:235-236metric_names.get(type(check), check.name)missesCustomSQLCheckemitted name ·phlo_pandera/plugin.py:416-419·QueryCheckRunnerapplies the routing ref but_load_datadoes notphlo_pandera/decorator_helpers.py:70-72· ✅ Addressed in b296dcc🪧 More tips
npx skills add coldteadotai/pr-lens, then tell your coding agent: "Diagram the change you just made with PR Lens and attach it to the pull request."npx @coldtea/pr-lens-cli analyze --base origin/mainon a branch, thennpx @coldtea/pr-lens-cli render .pr-lens/graph.json. Same lenses, your own model key, before the pull request existsrenderreads.github/pr-lens.ymland applies your renames, exclusions and lane pins at draw timegithub.comment.collapsed: truein.github/pr-lens.ymlto fold the comment behind one View architecture and data flow row. Drawing still runs as beforegithub.draw: on-demandin.github/pr-lens.ymland PR Lens stops drawing pull requests on its own. Comment@pr-lens drawon a pull request when you want that one drawn.github/workflows/pr-lens.ymlwithcoldteadotai/pr-lens/packages/action@v0and your model provider's key as itsapi-keyto run PR Lens from your own CI. Any/chat/completionsendpoint worksgithub.description: falsein.github/pr-lens.ymlto stop PR Lens adding buttons to your PR descriptionsThanks for using PR Lens! It's free for OSS. If it helped, a star helps other teams find it.