#1144

fix(ci): verify every first-party image signature before failing the rescan

Openiamgpwants to merge intofrom

Goal and scope

Follow-up to #1171 and #1178. The scheduled Container Rescan ("containers / published digest rescan") has failed every day. Its signature step ran cosign verify inside a loop under set -euo pipefail, so the first failed verification ended the step. Later first-party images were never checked, and each run reported only one missing signature. In run 41044371679, verification stopped at phlo-minio:0.17.1, so phlo-minio:0.17.2 and phlo-observatory:0.18.0 were never verified.

Goal: verify every first-party rescan target and report all failures in one run.

Non-goals: no change to which images are selected, to the certificate identity or issuer, or to the Trivy step. No waivers or skips are added.

Change map and risk

This is a behaviour change in .github/workflows/container-rescan.yml, limited to the "Verify signatures for every first-party rescan subject" step:

  • cosign verify is wrapped in if !. A failure writes ::error::Signature verification failed for <image>@<digest>, records failed=1 and moves on to the next target.
  • The step ends with exit "$failed", the same shape as the Trivy step.

Risk is low. The step still fails whenever any signature fails, and it now names every failing image. A malformed digest still fails the step straight away through the existing digest check.

Expected state after merge: the rescan of the published 0.18.0 release stays red until the next release rebuilds images.

  • phlo-minio:0.17.1 was published before publish-minio.yml gained signing, so it has no signature.
  • The 0.18.0 phlo-api and phlo-dagster images bundle uv < 0.13.4 (fixable CRITICAL GHSA-hq85-5cpj-mr62). main already pins uv 0.13.4.

Verification

  • Ran the step's script, extracted from the workflow YAML, with a stub cosign and five fake targets: four first-party and one from docker.io. The stub forced failures for phlo-minio:0.17.1 and phlo-observatory:0.18.0.

    • All four first-party targets were verified, MinIO against the publish-minio.yml identity and the others against release-promotion.yml.
    • The docker.io target was skipped.
    • Both failures were reported as error annotations, and the step exited 1.
  • All-pass control: with a stub that always succeeds, the step exited 0.

  • make actionlint: passed.

  • make zizmor: no findings.

  • Not checked: a live run against GHCR. The next scheduled or manual rescan will show it.

  • Regression test: test_rescan_verifies_every_signature_before_failing in tests/tooling/test_container_security_workflows.py runs this step's script against a stub cosign with three first-party images. In the failing case the stub rejects two of them, and the test asserts that all three were checked, that both failures are named in ::error:: annotations, and that the step exits 1. An all-pass control asserts exit 0. Against the pre-change step from main, the failing case fails because only two images were checked; with this change both cases pass.

Recovery and maintainer decisions

Rollback: revert this commit. Nothing is published or mutated. Pushing and opening this PR were approved by the maintainer.

Activity

302
coldtea-pr-lens[bot]8h ago

NOTE

Nothing here is code, only docs, data or images, so PR Lens left this pull request undrawn. Comment @pr-lens draw to draw it

github.comment.notice: false in .github/pr-lens.yml turns this note off

phlo-agent[bot]8h ago

Static review of f18fd0f (only .github/workflows/container-rescan.yml changes).

No correctness, security, or compatibility defect found in the shell change. The if ! cosign verify wrapper takes that command out of errexit scope, failed=1 survives the loop because the loop is fed by a redirection rather than a pipeline, and exit "$failed" yields the aggregate status (.github/workflows/container-rescan.yml#L143-L163). The later Trivy step still runs through !cancelled() && steps.fleet.outcome == 'success', and the report upload is always(), so a failed signature no longer hides the rest of the rescan.

Testing gap: the change has no regression test, and the PR's justification for that is inaccurate. It states the repository "has no test harness for workflow shell steps", but tests/tooling/test_container_security_workflows.py loads container-rescan.yml, extracts run: scripts, and executes them against stub binaries. test_rescan_includes_independent_minio_before_a_phlo_release runs a step with a stub gh (L294-L355), and test_rescan_selects_latest_stable_release_across_pages (L146-L176) was added by #1178 as the regression test for the previous fix in this same workflow. The verification matrix also asks workflow changes to exercise workflow-specific fixture paths (L25).

A test that fails on the pre-change step and passes now is small: write a manifest with three first-party targets, stub cosign to fail for two while recording every invocation, and assert exit 1 with all three attempts; add an all-success control asserting exit 0. Without it, a future edit that restores a bare cosign verify would silently return to first-failure-only reporting with green CI.

Still unverified: the PR check suite at f18fd0f was queued when this review was written, so CI has no verdict yet, and the PR correctly notes that no live GHCR rescan has exercised the change. The next scheduled or manual rescan remains the proof against real signatures.