Goal and scope
Make an event-time freshness promise executable instead of treating successful ingestion as evidence that source events are recent. First of three stacked core improvements informed by the SSP data-engineering vault.
Change map and risk
- Add keyword-only
SLA.freshness_columnand optional timezone-awarefreshness_reference_time. - Compile explicit SLAs through existing neutral rules and Pandera execution. Preserve legacy SLA serialisation and ingestion-time behaviour.
- Fail explicit promises when timestamps are missing, unusable, or absent. Keep the age boundary inclusive.
- Document adoption and add a runnable IoT freshness demonstration.
Original verification
- Original head: focused contract, neutral-provider, Pandera helper and decorator suites: 103 passed, one integration test deselected.
- Same demonstration before/after: 30-hour-old events pass legacy ingestion freshness but fail the explicit two-hour event-time promise. Fresh and exact-boundary data pass; missing, invalid, null, empty and over-boundary cases fail, with reopened SQLite evidence.
- Additional integrated Trino/Iceberg/PostgreSQL execution verifies stale/empty/over-limit Dataset publication blocks and exact-boundary publication succeeds. This is publication-policy evidence, not physical Nessie promotion.
- Reproduce locally:
uv run --locked python examples/lakehouses/iot-telemetry/scripts/freshness_demo.py --output /tmp/iot-freshnesswith a fresh output directory. See the IoT README for author declarations. - Full integrated baseline: 7,417 passed, four skipped; core regression: 447 passed; real storage regression: 101 passed. These broader results cover the final stack, not this isolated head.
- Links and generated references pass. The final combined production documentation build exceeded the development orb's 6 GiB memory limit.
- New feature: boundary and missing-data regression coverage is included; no existing bug-fix issue is claimed.
Review correction
Empty-data handling is now per check: an explicit freshness promise still fails on empty input, but neighbouring legacy checks retain their skip behaviour. The extended public declaration test failed before the fix; the decorator suite now passes all 72 selected tests. The original head also passed GitHub CI, including documentation rendering.
Post-correction integrated stack: make check passes with 7,419 tests, four skipped and 194 deselected; make test-core-regression passes all 447 tests. These are combined-stack results, not a separate full baseline at this isolated head.
Recovery and maintainer decisions
Opt-in; no migration, deployment, or shared-setting change. Remove explicit declarations before downgrading. A source revert does not undo data already published. Freshness does not imply completeness or reject future timestamps. Maintainer review is pending; no independent reviewer or release qualification is claimed.