#2084
skill-creator: run_eval threshold logic is asymmetric: should_trigger and should_not_trigger use opposite pass conditions
anthropics/skills/blob/main/skills/skill-creator/scripts/run_eval.py
A higher --trigger-threshold makes positive queries harder while also making negative queries easier to pass. This is because the pass rule is >= threshold for positive examples and < threshold for negative examples, which effectively expands the acceptance band for negatives as the threshold increases.
Being able to specify separate thresholds for should and should-not trigger queries would help.
Activity
0No activity yet.