#1157
fix(run-evidence): close SQLite connections in sensors, API and tests
Why
ADR 0053 decision 5 requires Python 3.13 core and package test lanes. On 3.13 a SQLite connection that is never closed raises ResourceWarning: unclosed database when it is garbage-collected. filterwarnings = ["error"] turns that into a failure, and because it happens at garbage collection, a different test fails on each run. On main this affects about 10 core tests per run and tests in phlo-api, phlo-dagster, phlo-dlt, phlo-polaris and phlo-iceberg. This is the first of three PRs before the 3.13 CI lanes.
Production fixes
_SqlRunEvidenceStoresupportswith, which calls the existingclose().phlo_dagster.wap_sensorscloses the store fromdefault_run_evidence_store()after it reads quality results and after it reconciles a promoted run. Before this change, each call leaked one SQLite connection._idempotency_connection()inphlo_api.api.operation_controlscloses its connection if schema setup raises. All six callers already close it on the normal path.- The Observatory read-model cache serialises the value before it opens a connection. A value it cannot serialise no longer leaks a connection.
Test fixes
- A new autouse fixture in the root
conftest.pytracks everySQLiteRunEvidenceStoreand pyicebergSqlCatalog/InMemoryCataloga test creates. When the test ends, it closes each store and disposes each catalog engine. This covers about 110 store constructions and the catalog fixtures without editing each call site, including stores that hooks create indirectly. - Tests that open
sqlite3.connect(...)directly now wrap it inclosing(...).with sqlite3.connect()alone commits but does not close. - The WAP sensor tests wrap their injected store in
nullcontextso the shared store stays open.
Verification
- Python 3.13.14 with
-W error::ResourceWarning:- Core: 3420 passed. The one failure,
test_ci_test_partition, is the clickhouse-connect'u'warning in its subprocess, which the next PR fixes. - phlo-api 1368, phlo-dagster 545, phlo-iceberg 391 and phlo-polaris 40 passed.
- A connection tracker found no unclosed run-evidence, catalog or API connections. The one remaining unclosed connection is inside
coveragewhen it reads a deliberately corrupt data file intest_unreadable_coverage_data_blocks, which passes.
- Core: 3420 passed. The one failure,
- Python 3.12:
- Core: 3421 passed, and
make test-core-regressionpassed 447. - phlo-api, phlo-dagster, phlo-polaris and phlo-iceberg pass unchanged.
- phlo-dlt fails only
test_csv_upload.py::test_candidate_staging_supplies_required_provenance_to_the_real_iceberg_schema, an order-dependent flake that also fails on main. It is tracked separately.
- Core: 3421 passed, and
make checkpasses.
ea64ac2, the branch has moved onArchitecture
Data flow
Review
1 finding · reviewed
ea64ac2🟠
from pyiceberg.catalog.sql import SqlCatalogruns for every test ·conftest.py:121Correctness ·
conftest.py:121The autouse
close_test_sqlite_connectionsfixture importspyicebergduring setup. pyiceberg is only in thedevdependency group, not[project.dependencies], so any core-onlypytest testsrun errors every test withModuleNotFoundErrorbefore it starts.Fix: Import
SqlCataloglazily inside the fixture and skip its release when pyiceberg is not installed, the way_PHLO_TESTING_FIXTURE_MODULESand theiceberg_catalogfixture already guard optional packages.🤖 Prompt to fix review comments
🪧 More tips
npx skills add coldteadotai/pr-lens, then tell your coding agent: "Diagram the change you just made with PR Lens and attach it to the pull request."npx @coldtea/pr-lens-cli analyze --base origin/mainon a branch, thennpx @coldtea/pr-lens-cli render .pr-lens/graph.json. Same lenses, your own model key, before the pull request existsrenderreads.github/pr-lens.ymland applies your renames, exclusions and lane pins at draw timegithub.comment.collapsed: truein.github/pr-lens.ymlto fold the comment behind one View architecture… row. Drawing still runs as beforegithub.draw: on-demandin.github/pr-lens.ymland PR Lens stops drawing pull requests on its own. Comment@pr-lens drawon a pull request when you want that one drawn.github/workflows/pr-lens.ymlwithcoldteadotai/pr-lens/packages/action@v0and your model provider's key as itsapi-keyto run PR Lens from your own CI. Any/chat/completionsendpoint worksgithub.description: falsein.github/pr-lens.ymlto stop PR Lens adding buttons to your PR descriptionsThanks for using PR Lens! It's free for OSS. If it helped, a star helps other teams find it.