Incident 2026-10-18: checksums.yaml-only re-pin (7368ba66) broke curl|bash bootstrap integrity for ~3 min; restored in 9956dd9c

Summary

On 2026-10-18, commit 7368ba66 changed checksums.yaml alone (it re-pinned atuin and opencode). The bootstrap's internal integrity ledger scripts/generated/internal_checksums.sh also pins the sha256 of checksums.yaml, so while main pointed at 7368ba66, every curl | bash ACFS install stopped during bootstrap:

ERROR: INTEGRITY: checksums.yaml checksum mismatch
ERROR: Internal script integrity check failed: 1 file(s) missing, unsafe, or modified
✖ Bootstrap archive failed its internal executable integrity contract

9956dd9c restored the exact f13da7b3 bytes of checksums.yaml, and main bootstraps again. This issue records what happened and the prevention that is still missing. Filed by the release-campaign agent (StormyMill) that made the change.

Timeline (UTC; commit and push times from the pushing clone's reflog)

Exposure window: main and master resolved to 7368ba66 from about 19:24:48 to about 19:27:54 (roughly 3 minutes), measured from my own pushes. GitHub-side propagation and any caching are not measured, so the real window is not known exactly. The nightly acfs-update window was not hit.

Restoration proof

• git diff f13da7b3 9956dd9c is empty.
• Raw checksums.yaml at 9956dd9c has sha256 6582c752935b3dbf5e6168afca561a37e8d00535532685b19b0899750fc241ba, the value the ledger pins.
• Real one-liner from main in a clean ubuntu:24.04: Bootstrapping ACFS archive (9956dd9c…), ✓ Internal script integrity verified (121 scripts), ✓ Verified: zoxide, ACFS_RC=0, zoxide 0.10.0. The release coordinator ran the same check independently and got the same result.
• This proves the bootstrap path and one verified-installer module. It does not prove every module.

Affected-module status after the restore:

• tools.atuin and agents.opencode are back on their f13da7b3 pins. They still fail verification against the current upstream bytes, as they have since the monitor deferred them. That deferral is pending human authorization in #466 (digest 6b9439ca742656dc90d1ce8038620577b0ffde628ea3af9cfc3df58c12d6174b). I recomputed this digest with the monitor's external_authorization_digest formula and it is still current.
• All other pins matched their live installers at 18:50 UTC (46/48 OK; only these two drifted).

Root cause

1. A checksum change that bypassed the generator. I followed AGENTS.md "Verified Installer Checksum Discipline": generate a candidate with ./scripts/lib/security.sh --update-checksums, review the diff, and replace checksums.yaml with it. That section does not say that checksums.yaml is itself pinned by scripts/generated/internal_checksums.sh and apps/web/lib/generated/manifest-modules.ts (checksumsYamlSha256). Both must be regenerated, for example through packages/manifest bun run generate, the way the checksum monitor's commits do it (0e8de312 touches all three files).
2. The ledger guard did not run. scripts/hooks/pre-commit has an "Internal checksum ledger guard" that would have rejected this commit, but it is not active in a fresh clone. My clone was a plain git clone, with no core.hooksPath set and no hook installed.
3. External changes went around the human gate. atuin and opencode are External installers. scripts/checksum-monitor-local.sh defers them until an owner-only, non-symlink file $STATE_DIR/authorized-checksum-change (default ~/.local/state/acfs-monitor/) contains exactly authorize:<digest> (authorization_is_valid, prepare_checksum_candidate). My manual commit published them without that gate. Upstream content had been reviewed (atuin v18.23.0 → v18.24.1 changes only version strings and asset hashes; opencode is the v1-branch installer at 3be03959) and tested with real installs, but that review does not replace the owner authorization.

Missing prevention: proposals for the owner, nothing applied

• AGENTS.md "Verified Installer Checksum Discipline": state that checksums.yaml is ledger-pinned, that a pin change must regenerate internal_checksums.sh and manifest-modules.ts in the same commit, and that External tools go through the monitor's authorization file.
• Make the ledger guard hard to miss. For example, a scripts/ entry point that checks the ledger before push, and an AGENTS.md instruction to set git config core.hooksPath scripts/hooks in every clone, including agents' scratch clones.
• A test that fails when sha256(checksums.yaml) does not equal the [checksums.yaml] entry in scripts/generated/internal_checksums.sh or checksumsYamlSha256 in apps/web/lib/generated/manifest-modules.ts, run by the repo's local test entry point and not only by Actions. tests/vm/bootstrap_offline_checks.sh already exercises the bootstrap from a local archive and could be the pre-publication gate for pin changes.

Evidence (trj, /data/relwave/oct18-lane1/logs/)

• acfs-oneliner-e2e-2404.log: the failing bootstrap against 7368ba66
• acfs-oneliner-e2e-restore.log: the passing bootstrap against 9956dd9c
• acfs-thirdparty-probe-1922.log, acfs-thirdparty-probe-1925-arm64-2204.log: security.sh probes (old pins refused; new pins install atuin 18.24.1 and opencode 1.19.3 on 24.04 amd64/arm64 and 22.04)
• acfs-checker-validate-candidate-1927.log: automated_flywheel_setup_checker 0.1.0 validate, 48/48 reachable and matched for the candidate pins