The edge case
ci-score refuses to grade repositories whose visible workflows do no project build or test — only automation (bots, releases, triage). That refusal exists for a good reason: the rubric's registry decision log records a large public repo (anthropics/claude-code) whose visible workflows are exclusively issue-triage bots and release automation; grading it produced a real-looking number that described nothing about the project's actual CI.
The refusal predicate reads workflow YAML only: it looks for a test signal (test-like job or test-runner command), a build signal (build tool/config/command), or a dependency-install command. Any one signal scores the repo — the bias is deliberately toward scoring.
The gap: a repository whose entire CI is delegated to a repo script — e.g. a single job whose only step is run: ./ci.sh — shows none of those signals in YAML, even though ci.sh runs a full install + build + test pipeline. Such a repo is refused as automation-only: a false refusal.
Why the obvious mitigation was rejected (so far)
Reading the referenced scripts' contents to find test/build signals would un-refuse script-wrapped repos — but it would also un-refuse the motivating case class: automation-only repos whose bot scripts contain incidental words like make or install their own tooling. The predicate's honesty currently depends on not interpreting script contents.
Possible directions (none chosen)
- Accept and document — the refusal message already invites a mailto; script-wrapped CI is rare in the calibration population (0 of 31 repos).
- Narrow script-reading — only scripts directly invoked by a
run:step, only searching for unambiguous test-runner invocations (pytest,go test, …), never counting bare words; would need its own calibration pass. - A user override — let the user assert "this repo's CI is real" and score with a disclosure noting the assertion.
Filed to track the owner decision; the current shipped behavior is (1).
Activity
0No activity yet.