#120

Qualify one finite security-evolution workflow before expansion

Openbart0nautopened 1mo ago

Product decision

Decide whether offline width-one security evolution improves one useful security workflow enough to justify its full cost and latency. This is a finite go/no-go evaluation, not a commitment to build a general optimizer or keep expanding research machinery.

The controller engineering is implemented in #195 and the earlier provenance, retention, holdout and accounting work. Corrected source selections, negative-control isolation, explicit budgets, complete declared final reserves and zero-model controls are present. These checks establish controller behavior only. They do not establish model-backed vulnerability detection, optimizer benefit, actual charges or deployed performance.

Remaining work, only when prioritized and funded

  • Choose one product-relevant workflow and independently adjudicate a small vulnerable/fixed/clean corpus. Advisory version labels plus a literal sink match are insufficient vulnerability proof.
  • Predeclare one primary decision, a finite total spending/time cap, meaningful quality/false-alarm/retention tolerances and source/runtime/provider identities. Record unsupported identities or measurements as unknown; do not manufacture qualification fields.
  • Compare the frozen baseline with width-one evolution and, where useful, budget-matched extra baseline attempts. Include rejected proposals, failures, search/build/evaluation work and final evaluation in the totals. Keep negative and final controls out of proposer feedback.
  • Report case-level outcomes and uncertainty, all model/compute charges or unresolved billing, and end-to-end latency. Report tail latency only when supported; a p95 requirement is not a reason to expand an otherwise finished finite pilot.
  • Make a go/no-go decision. A negative or inconclusive result within the fixed cap is a valid outcome: retain the simpler baseline and close the decision. Do not repeat or expand merely to obtain a positive result.

No paid execution is authorized by this issue. Production defaults, scan prompts, proposal width and ordinary scan call counts stay unchanged. No automatic promotion or deployment follows from a study result.

Consolidated engineering qualification

The implemented safeguards from #37, #38 and #40 do not need separate architecture work. Their empirical acceptance is retained here: measure retention/adaptation and false alarms, bind a genuinely pinned worker/workflow and complete usage/charges, preserve independent family splits, and distinguish untouched final outcomes from reused selection/canary observations. Exposure accounting is not proof of adaptive validity or generalization.

#41 already supplies the phase-one durable ledger and subprocess recovery controls. Before unattended operation, qualify actual CLI/VM/provider interruption at dispatch, receipt and promotion boundaries with authoritative charge reconciliation and no blind replay. The initial finite supervised pilot does not require a new scheduler or unattended platform. Feedback wakeups and multi-day scheduling stay deferred until observed product value justifies them.

Deferred hypotheses

#39 (archive-parent selection), #121 (compact development feedback) and #122 (wider same-parent proposals) have bounded experimental controller implementations, but no demonstrated product benefit. They are consolidated here as deferred hypotheses, not active delivery commitments. Reopen only after the basic workflow is qualified, an observed bottleneck makes the hypothesis relevant, and a separate finite comparison is explicitly prioritized. Do not combine summaries, wider proposals and archive-parent selection in an initial pilot.

Independent curation and paid/model-backed qualification remain unperformed. Deterministic null controls are not evidence that the model-backed hypotheses are either useful or useless.

Backlog review and validation: #212, docs/qualification/issue-triage-2026-10-27.md.

Activity

0

No activity yet.