#2084

skill-creator: run_eval threshold logic is asymmetric: should_trigger and should_not_trigger use opposite pass conditions

Openbenharvey-bisopened 4d ago

anthropics/skills/blob/main/skills/skill-creator/scripts/run_eval.py

A higher --trigger-threshold makes positive queries harder while also making negative queries easier to pass. This is because the pass rule is >= threshold for positive examples and < threshold for negative examples, which effectively expands the acceptance band for negatives as the threshold increases.

Being able to specify separate thresholds for should and should-not trigger queries would help.

Activity

0

No activity yet.