DEP-91 (4/5): weekly base-OS rebuild of hoophq/hoopagent

• Add a scheduled workflow to rebuild the minimal agent image OS layer weekly.
• Fetch and checksum-verify the latest released agent binaries; do not rebuild the app.
• Publish :latest and a dated snapshot only if both arches pass OS-layer Trivy gates.

graph TD
  T(["Schedule / workflow_dispatch"]) --> C["Checkout repo"] --> R["Resolve latest release + download tarballs"] --> V{"SHA256 verify"} --> BA["Build cand amd64 (load)"] --> GA{"Trivy OS gate amd64"} --> BB["Build cand arm64 (load)"] --> GB{"Trivy OS gate arm64"} --> RG["Report Go-layer CVEs (non-blocking)"] --> P["Build+push multi-arch tags"] --> H{{"Docker Hub"}}
  R --> S{{"releases.hoop.dev"}}
  V -- "fail" --> X(["Abort (no publish)"])
  GA -- "fail" --> X
  GB -- "fail" --> X

  subgraph Legend
    direction LR
    _evt(["Workflow trigger"]) ~~~ _step["Workflow step"] ~~~ _dec{"Gate / decision"} ~~~ _ext{{"External system"}}
  end

The following are alternative approaches to this PR:

1. Rebuild from source on schedule (cut a ‘nightly’ release)

• ➕ Ensures Go dependency/toolchain CVEs can be fixed without waiting for a manual release
• ➕ Produces a fully fresh artifact (binary + OS) each run
• ➖ More moving parts (toolchain, reproducibility) and longer build times
• ➖ Blurs the line between ‘release artifacts’ and ‘scheduled maintenance’, increasing risk of unintended behavior changes

2. Only rebuild/push on base-image digest change

• ➕ Avoids unnecessary rebuilds when upstream OS layers haven’t changed
• ➕ Can reduce registry churn and CI usage
• ➖ Requires additional logic to detect and compare base image digests reliably
• ➖ Still needs the same gating/publish logic; complexity savings are limited

3. Use registry-side automated rebuilds (if supported)

• ➕ Potentially simpler CI footprint (less build infra in GitHub Actions)
• ➖ Harder to enforce multi-arch + per-arch OS-layer gates before publish
• ➖ Less transparent/auditable than repo-defined workflows

Recommendation: The chosen approach is a good fit: it preserves the release binary (immutability of app bytes), refreshes only the OS layer for patch cadence, and enforces OS-only gates per architecture before publishing. Keeping Go/library CVEs report-only avoids an OS patch pipeline being wedged by a lagging application release.

Documentation (1) +20 / -0

custom-agent-image.mdDocument minimal image tag semantics and weekly OS patch cadence +20/-0

Document minimal image tag semantics and weekly OS patch cadence

• Adds documentation describing the new weekly repatch behavior for hoophq/hoopdev-minimal and clarifies when to use :latest vs dated snapshot vs immutable release tags. Recommends digest pinning for fully reproducible builds and clarifies that Go-layer CVEs are addressed via releases, not weekly OS rebuilds.

docs/custom-agent-image.md

Other (1) +178 / -0

minimal-image-rebuild.ymlAdd weekly minimal-image rebuild workflow with per-arch Trivy OS gate +178/-0

Add weekly minimal-image rebuild workflow with per-arch Trivy OS gate

• Introduces a scheduled/manual GitHub Actions workflow that downloads and checksum-verifies the latest released agent binaries, rebuilds the minimal Docker image for amd64/arm64, and enforces fixable CRITICAL/HIGH OS-layer vulnerability gates via Trivy. On success, publishes multi-arch :latest and a dated :<release>-<YYYYMMDD> tag; also emits a non-blocking report for Go/library vulnerabilities.

.github/workflows/minimal-image-rebuild.yml

Description

A new comment mentions DEP-91 but does not start with the ticket ID, violating the required Linear
ticket comment prefix format. This makes automated linkage/search for ticket references unreliable.

Code

.github/workflows/minimal-image-rebuild.yml[2]

+# (hoophq/hoopdev-minimal). Closes DEP-91 point 4: "weekly base-OS patches".

Relevance

●● Moderate

Mixed precedent on Linear-ID comment prefix: accepted in #1642 but same request rejected in #1655.

PR-#1655 PR-#1642

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 2362449 requires that any comment mentioning a Linear ticket begin with the ticket
ID. In the new workflow file, line 2 mentions DEP-91 but the comment begins with
(hoophq/hoopdev-minimal)..., not DEP-91:.

Rule 2362449: Prefix external issue IDs in code comments that reference Linear tickets .github/workflows/minimal-image-rebuild.yml[1-2]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A new comment references a Linear ticket (`DEP-91`) but the comment text does not begin with the ticket ID token as required.

## Issue Context
The compliance rule requires any comment referencing a Linear ticket to start with the ticket ID in the format `[A-Z]+-[0-9]+` immediately after the comment delimiter.

## Fix Focus Areas
- .github/workflows/minimal-image-rebuild.yml[1-3]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

The workflow uses sha256sum -c --ignore-missing without asserting that the downloaded tarballs
were actually verified; if checksums.txt ever lists just basenames (a format supported by the
repo’s checksum parser) or otherwise doesn’t match dist/binaries/..., the verification can
silently skip all relevant entries and still succeed, allowing unverified binaries to be
republished.

Code

.github/workflows/minimal-image-rebuild.yml[R79-82]

+          # checksums.txt paths are ./dist/binaries/...; verify from the repo root.
+          # --ignore-missing skips the other-platform entries but still fails if
+          # a staged tarball mismatches or none were verified.
+          sha256sum -c --ignore-missing dist/binaries/checksums.txt

Relevance

●● Moderate

Security-hardening suggestion; no close repo precedent for enforcing non-no-op sha256sum
verification in workflows.

PR-#1629

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The workflow claims verification “fails closed” but uses --ignore-missing and does not check that
any tarball was verified. The repo’s release checksum generation and client-side verification logic
indicate checksum entries may include paths or just basenames, making path-dependent sha256sum -c
brittle when paired with --ignore-missing.

.github/workflows/minimal-image-rebuild.yml[57-85] Makefile[178-184] Makefile[297-301] client/upgrade/installer.go[155-177] client/upgrade/installer_test.go[113-120]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The weekly rebuild workflow runs `sha256sum -c --ignore-missing` but does not ensure that the specific staged tarballs were verified (i.e., at least one/each expected tarball produced an `OK`). With `--ignore-missing`, a checksum file that doesn’t reference the staged paths can result in a no-op check that still exits 0.

## Issue Context
The repo’s own checksum parsing logic matches by basename (not full path), so the checksum file format is intentionally flexible; relying on path-matching plus `--ignore-missing` is brittle and can defeat the intended “fail closed” behavior.

## Fix Focus Areas
- .github/workflows/minimal-image-rebuild.yml[57-85]

### Concrete fix idea
- Extract the checksum lines for the two exact tarballs you downloaded (x86_64 and aarch64) using `grep` on `checksums.txt` (matching by basename), write those to a temporary file, then run `sha256sum -c` **without** `--ignore-missing` on that temp file.
- Additionally, assert both expected artifacts were found in the checksum file before proceeding (e.g., count matches == 2).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

The workflow attempts Docker Hub login using secrets.DOCKER_LOGIN/secrets.DOCKER_PASSWORD
without an explicit preflight check, so misconfiguration yields a less actionable failure message
than a dedicated “missing secret” error.

Code

.github/workflows/minimal-image-rebuild.yml[R52-55]

+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKER_LOGIN }}
+          password: ${{ secrets.DOCKER_PASSWORD }}

Relevance

●●● Strong

Team previously accepted failing early when required secrets are missing in workflows for clearer
errors (#1595).

PR-#1595

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The workflow directly passes Docker Hub secrets into the login action without validation. A prior
accepted workflow fix in this repo added early secret checks to improve failure clarity for
misconfigured secrets.

.github/workflows/minimal-image-rebuild.yml[45-56] PR-#1595

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Add an explicit early step that fails with a clear message if required Docker Hub secrets are missing/empty, improving diagnosability of scheduled/manual rebuild failures.

## Issue Context
This repo has previously accepted adding early secret validation to avoid confusing failures later in workflows.

## Fix Focus Areas
- .github/workflows/minimal-image-rebuild.yml[45-56]

### Concrete fix idea
Insert a step before `docker/login-action`:
- `if: ${{ secrets.DOCKER_LOGIN == '' || secrets.DOCKER_PASSWORD == '' }}`
- Emit `::error::DOCKER_LOGIN/DOCKER_PASSWORD not set` and exit 1.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools