A speech judge against real voices: chimera compare tsp

chimera compare tsp says the Harvard sentences that 23 real speakers read (the TSP speech database, McGill; 12 women and 11 men at 48 kHz) and compares the synth with them, phone by phone. Whisper's word error rate says whether a sentence is understood. This judge says which sound is wrong and by how much. The spec is docs/speech-comparison.md.

How it works

• Rendering. The synth says each sentence exactly as render text does (refactored into render::say; the output is byte-identical).
• Timing. The Montreal Forced Aligner times both voices with the same model and settings, pinned and made deterministic: no dither, no speaker adaptation, each utterance its own speaker. The recordings' alignment is committed; the renders are aligned on each run. The planner's own segment times aren't used: MFA puts the synth's vowels 29 ms shorter than the plan's segments.
• Matching and measuring. Phones are matched by Levenshtein, and both voices are measured by the same code.
• Scored: consonant levels and bursts, sibilant centroids, and timing, in units of how much the real speakers differ from each other.
• Diagnostics only: mel-cepstra, formants, vowel bands and H1−H2, voicing, f0 range. The voice terms call batch 1 worse, while the owner's listening preferred it 7 to 0, so they don't score until they're checked against listening.
• Verdict. --against counts only scored terms. It breaks a term that rises by more than 0.5 or loses 5 % of its tokens. It refuses runs with another judge version, recording set, alignment, model, MFA version or settings.

Baseline (main, after batch 2)

• Score: 1.583 on the training speakers' sentences, 1.706 held-out. A real speaker scored the same way gets 0.277.
• The batches: 1.994 before batch 1, 1.797 after it, 1.583 after batch 2.
• Largest leads:
  • The female /ʃ tʃ/ are 16–17 dB too quiet (/ʃ/ −23 dB re her stressed vowels, women −7).
  • Her /s z/ centroids are 4.0–4.7 kHz (women 7.1–7.4).
  • /k/ and /dʒ/ are short (/dʒ/ half what speakers give it).
  • /ð v/ are 7–11 dB too loud.
• A diagnostic: after batch 1 the voice lacks 1–4 kHz presence (female 1–2 kHz −16 dB re 0–8 kHz, women −11). That is a spectral shape, not a tilt.

Data and licences

• TSP database. Simplified BSD (the report is CC BY 4.0). The derived sentence list and alignment (290 kB gzipped) are committed, with the notice in tools/tsp/README.md. The audio is fetched by tools/tsp/fetch.sh (sha256 checked).
• Aligner. MFA (MIT) and its english_us_arpa v3.0.0 model (CC BY 4.0) are installed by tools/tsp/setup.sh, not redistributed.
• Wrong recordings. Four recordings say another sentence than their label. Whisper found them (tools/tsp/check_text.py), and they are excluded. Whisper gets 2.7 % of words wrong on the real recordings, against about 37 % on the synth.

Reviews

• Design review (Codex). Twelve findings. The main changes: MFA timing for both voices instead of the planner's times, scoring only the validated measures, per-speaker-set synth values, per-term token-support guards, affricate and burst windows, and percentile terms with worst-token lists for pops and squeezed vowels. Declined: word-constrained matching (0 of 32,766 matched pairs cross a word), and per-instance verdict recomputation (the synth's token set is fixed by the G2P and the deterministic aligner).
• Code review (Codex). Four findings, all fixed in the last commit:
  • A missing measurement cost 0 (f64::max(NaN, 0)).
  • The verdict compared only the recording count.
  • The aligner version and settings weren't recorded.
  • A relative dataset path broke the symlinks.

Checks

• cargo test --workspace, bun test and bun run typecheck pass.
• New unit tests cover matching, the ARPAbet round trip, the dictionary's reduced function words, missing costs and the verdict.
• Two full runs are identical (verdict +0.000).

Not done

• No batch uses this judge yet. Whether it decides the next batch, with Whisper as a guard, is the owner's call.
• How a change in score maps to what a listener hears is unmeasured.

🤖 Generated with Claude Code