DEP-91 (5/5): harden image release integrity

• Verifies hand-installed binary archives (kubectl, awscli, sqlcmd) against reviewed SHA-256 pins,
 failing closed on any mismatch.
• Pins all GitHub Actions and Trivy invocations to immutable commit SHAs/digests instead of mutable
 tags.
• Restructures hoopdev-minimal weekly rebuild and release publishing to gate each per-arch candidate
 on Trivy before any push, avoiding partial multiarch manifests.
• Adds a GitHub Release-asset-based checksum trust anchor for the weekly minimal-image rebuild,
 falling back to reviewed compatibility pins for legacy releases.
• Makes the SBOM/manifest generator fail closed for required images/platforms, retries release-asset
 uploads, and reads bundled-tool versions from OCI labels on the exact shipped digest instead of the
 Dockerfile recipe.
• Adds report-agent-tools-versions.py to probe and pin the exact agent-tools base digest (with
 locked-down container) consumed by Dockerfile.dev builds.
• Adds unit tests for the SBOM generator's subprocess handling and wires them into pull-request CI.

graph TD
  A["Dockerfile.tools build"] --> B["verify-manual-download.py"] --> C["agent-tools image"]
  C --> D["report-agent-tools-versions.py"] --> E["Dockerfile.dev build"]
  E --> F{"Trivy gate per arch"}
  F -->|pass| G["Push candidate tag"] --> H["Compose multiarch manifest"]
  F -->|fail| I["Abort: no push"]
  H --> J["generate-sbom-manifest.py"] --> K["GitHub Release assets"]
  L["minimal-image-rebuild.yml"] --> M["GitHub Release digest / legacy pin fallback"] --> E

  subgraph Legend
    direction LR
    _svc([Process]) ~~~ _dec{Decision} ~~~ _ext[[Artifact/Store]]
  end

The PR's fail-closed, digest-pinned approach with per-arch gate-before-push is the standard and appropriate pattern for supply-chain hardening of container release pipelines; no meaningfully different architecture was worth considering given this is the final layer of an already-designed 7-part stack (DEP-91).

Enhancement (5) +475 / -260

minimal-image-rebuild.ymlRestructure weekly rebuild to trust GitHub Release digests and gate-before-push per arch +152/-56

Restructure weekly rebuild to trust GitHub Release digests and gate-before-push per arch

• Replaces the releases.hoop.dev checksum flow with GitHub Release asset digests as the primary trust anchor (falling back to reviewed compatibility pins for pre-DEP-91 releases), moves Docker Hub login after both arch candidates pass the OS-vuln gate, and composes/pushes multiarch tags from per-run candidate tags instead of rebuilding at push time.

.github/workflows/minimal-image-rebuild.yml

pullrequest.ymlRun new CI helper tests and add govulncheck for hsh-tunneld +15/-4

Run new CI helper tests and add govulncheck for hsh-tunneld

• Adds a unittest run for the SBOM generator helper script, pins checkout/setup-go actions by digest, disables credential persistence, and adds a govulncheck step scanning the separately shipped tunnel module.

.github/workflows/pullrequest.yml

release.ymlGate hoopdev-minimal per-arch, pin actions, and harden SBOM/release-asset steps +159/-133

Gate hoopdev-minimal per-arch, pin actions, and harden SBOM/release-asset steps

• Reworks hoopdev-minimal publish jobs to build-load-scan-then-push per architecture and only compose the multiarch manifest when both arches published; pins all actions/Trivy by digest; probes and labels the exact agent-tools base for hoopdev builds; makes SBOM generation and release-asset upload required/retried instead of continue-on-error; publishes Linux release archives as GitHub Release assets for the rebuild workflow's trust anchor.

.github/workflows/release.yml

Dockerfile.toolsVerify manual downloads against reviewed checksums and add provenance labels +55/-16

Verify manual downloads against reviewed checksums and add provenance labels

• Adds OCI labels recording bundled tool versions, copies the checksum manifest and verifier script into the build, and rewrites kubectl/awscli/sqlcmd download steps to verify each archive against reviewed SHA-256 pins before installing; cleans up the build-integrity helper files afterward.

Dockerfile.tools

generate-sbom-manifest.pyMake SBOM generation fail closed for required images and read tool versions from image labels +94/-51

Make SBOM generation fail closed for required images and read tool versions from image labels

• Removes Trivy Docker Hub credentials, requires gateway/fat-agent images and both platforms to be present or fails the job, reads bundled-tool versions from OCI labels on the exact scanned digest instead of parsing Dockerfile.tools, and enforces checked subprocess calls for Trivy.

scripts/ci/generate-sbom-manifest.py

Tests (1) +54 / -0

test_generate_sbom_manifest.pyAdd unit tests for SBOM generator subprocess handling +54/-0

Add unit tests for SBOM generator subprocess handling

• New test module verifying that the trivy() helper invokes subprocess.run with check=True and that the generic _run() helper defaults to check=False.

scripts/ci/test_generate_sbom_manifest.py

Documentation (2) +3 / -1

custom-agent-image.mdUpdate release-tag mutability documentation +1/-1

Update release-tag mutability documentation

• Clarifies that the ':<release>' tag can move on a release-workflow rerun rather than being permanently immutable, and recommends digest pinning for guaranteed byte stability.

docs/custom-agent-image.md

manual-binaries.tomlCross-reference new checksum manifest +2/-0

Cross-reference new checksum manifest

• Adds a comment pointing to the new agent-tools-checksums.sha256 file as the source of independent archive pins.

licenses/manual-binaries.toml

Other (8) +325 / -27

agent-tools-build.ymlPin actions/scanners by digest and defer Docker Hub login until after scan +27/-19

Pin actions/scanners by digest and defer Docker Hub login until after scan

• Replaces mutable action tags with digest-pinned versions, adds persist-credentials: false, pins Trivy by image digest, and reorders the arm64 job so Docker Hub login happens after the license/vuln scans instead of before.

.github/workflows/agent-tools-build.yml

minimal-image-build.ymlPin PR-smoke build actions and Trivy scanner +8/-6

Pin PR-smoke build actions and Trivy scanner

• Pins checkout, QEMU, buildx, build-push-action and Trivy to immutable digests and disables credential persistence on checkout.

.github/workflows/minimal-image-build.yml

pullrequest-release.ymlPin Trivy scanner image by digest +1/-1

Pin Trivy scanner image by digest

• Switches the license-scan Trivy invocation from a mutable tag to a pinned image digest.

.github/workflows/pullrequest-release.yml

Dockerfile.minimalPin base image by digest +1/-1

Pin base image by digest

• Adds a SHA-256 digest pin alongside the ubuntu:noble-20251013 tag.

Dockerfile.minimal

agent-tools-checksums.sha256Add reviewed checksum manifest for hand-installed tool archives +16/-0

Add reviewed checksum manifest for hand-installed tool archives

• New file listing reviewed SHA-256 digests for versioned kubectl, awscli, and sqlcmd archives across architectures, consumed by the Dockerfile.tools verification step.

licenses/agent-tools-checksums.sha256

legacy-release-checksums.sha256Add reviewed bootstrap checksum pins for pre-DEP-91 releases +5/-0

Add reviewed bootstrap checksum pins for pre-DEP-91 releases

• New compatibility manifest with reviewed digests for the two Linux release archives predating GitHub Release asset uploads, used only as a fallback in the weekly rebuild.

scripts/ci/legacy-release-checksums.sha256

report-agent-tools-versions.pyAdd script to probe and pin exact agent-tools base image metadata +190/-0

Add script to probe and pin exact agent-tools base image metadata

• New CLI that either reports bundled-tool versions from inside an agent-tools image or, with --prepare-build, resolves the pinned tag to an exact digest, runs the probe inside a locked-down container, and writes the digest-bound build arg and OCI labels to GITHUB_OUTPUT.

scripts/ci/report-agent-tools-versions.py

verify-manual-download.pyAdd checksum verification script for manually downloaded binaries +77/-0

Add checksum verification script for manually downloaded binaries

• New CLI that loads a reviewed SHA-256 manifest, computes the actual digest of a downloaded artifact, and fails closed on missing entries or mismatches using constant-time comparison.

scripts/ci/verify-manual-download.py

Description

Dockerfile.tools switched kubectl/aws/sqlcmd downloads to fail-closed curl --fail, but the new
download commands have no bounded retry policy, so transient CDN/network failures will now fail the
Docker build immediately and can cause avoidable CI/release flakiness.

Code

Dockerfile.tools[R129-132]

+    && curl --fail --location --silent --show-error \
+      --proto '=https' --tlsv1.2 \
+      "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${dpkgArch}/kubectl" \
+      --output "$artifact" \

Relevance

●●● Strong

Bounded curl retries reduce CI flakiness; similar Dockerfile.tools reliability hardening suggestions
were accepted.

PR-#1629

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new kubectl download uses curl --fail ... but no --retry, and the same pattern is used for
other manual downloads (AWS CLI zip and sqlcmd tarball). In contrast, the weekly rebuild workflow
already uses curl --retry 3 --retry-all-errors for integrity-critical downloads, showing an
established repo pattern for bounded resilience.

Dockerfile.tools[123-155] Dockerfile.tools[195-215] .github/workflows/minimal-image-rebuild.yml[121-134]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`Dockerfile.tools` now downloads kubectl/AWS CLI/sqlcmd using `curl --fail --location --silent --show-error` (good: fail-closed), but without `--retry` this makes the build brittle to transient network/CDN issues.

### Issue Context
This PR hardened integrity checks for manually downloaded binaries. To keep builds reliable, the download step should also be resilient. Other integrity-critical download paths in CI already use bounded retries.

### Fix Focus Areas
- Dockerfile.tools[127-155]
- Dockerfile.tools[205-213]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗