Nepali speech benchmark reveals gaps between automated scores and native listeners

Testing what sounds right

Automated speech evaluation can reward audio that is clean or easy to transcribe without capturing whether it sounds natural to native listeners. In the NepTTS-Bench preprint, Ashish Thapa, Rupak Ghimire and Lekhnath Sharma Pathak investigate that gap for Nepali text-to-speech, or TTS.

Their contribution is a common evaluation setting rather than a new speech generator. It combines pronunciation-focused sentences, blinded listener ratings, speech-recognition checks and a Nepali-specific quality predictor. The findings challenge simple automated rankings, but do not establish a single winning TTS system.

A shared listening test

The benchmark contains 205 sentences, including 77 minimal pairs: paired examples intended to isolate particular pronunciation contrasts. These target features such as aspiration, dental versus retroflex consonants, nasalization, consonant length and schwa deletion. Another 58 sentences cover cases including question intonation, emotional expression, homographs and code-mixed text.

The authors checked spelling, grammar and whether candidate pairs isolated the intended contrast, replacing or correcting some examples. They explicitly describe this as an engineering audit, not formal validation of the material as an account of Nepali phonology.

Although the wider corpus contains audio from 40 named systems or voices, the main statistical comparison covers ten TTS configurations and two human reference conditions. The paper reports 6,962 ratings from 180 native Nepali speakers for this panel. Listeners scored blinded audio from one to five, with randomized, balanced sampling across systems and separation between examples using the same sentence.

No clear winner among closely rated systems

Natural speech received a mean opinion score, or MOS, of 3.95, while human read speech scored 3.64. Among synthetic configurations, TingTing Asmita scored 3.55, Piper 3.50, and TingTing Subina and ElevenLabs v3 both 3.47. gTTS scored 2.60.

These averages need their uncertainty attached. Piper’s 95% confidence interval was 3.31–3.69, while ElevenLabs v3’s was 3.28–3.66. The authors report overlapping intervals across much of the higher-scoring group and say the data do not support naming one winner. They identify gTTS as clearly below the other systems.

Listener agreement was also low: Krippendorff’s alpha was 0.17. That limits how precisely this protocol can distinguish systems with similar perceived quality. The scores are comparative results under this listening setup, not universal properties of the products.

When automated rankings disagree

SCOREQ, a general-purpose audio-quality metric, ranked ElevenLabs v3 first but Piper eighth. In the paper’s human ranking, ElevenLabs placed fourth and Piper second among synthetic configurations. The authors interpret this divergence as evidence that signal quality alone misses aspects of Nepali pronunciation and delivery.

They also evaluated seven automatic speech recognition systems by transcribing generated audio and measuring character error rate against the input text. Lower error rates indicate closer transcription, but do not directly measure naturalness. Gemini Flash had the best Whisper-small error rate in the panel without receiving the highest listener score.

Across nine systems with automated coverage, SCOREQ’s Spearman rank correlation with human MOS was 0.40. Chirp 2 and MMS character-error metrics reached 0.60 and 0.58 respectively. The authors caution that this small comparison cannot establish a reliable distinction between those two recognition-based metrics.

Whisper large-v3-turbo produced extreme transcription hallucinations in some cases, distorting average error rates. The authors excluded it from the main automated comparison and recommend hallucination filtering for this model in Nepali round-trip evaluation.

A predictor with a transfer problem

NepaliMOS adapts an IndicWav2Vec audio encoder to predict listener scores. It uses audio alone, without reference text or speech-recognition input, and learns from mean ratings for individual clips.

With a random 90/10 utterance split within the evaluated panel, the authors report correlations of 0.59 at utterance level and 0.90 at system level. NepaliMOS was the only metric whose agreement with human system rankings survived their multiple-testing correction.

However, the random split includes systems represented during training. When the researchers instead withheld an entire TTS system, utterance-level correlation fell to 0.43, and held-out system means were not ranked reliably. These results support using NepaliMOS as an auxiliary signal, not an independent judge of unfamiliar providers.

The authors recommend a small human calibration set before evaluating a new provider. Wider, sparsely rated pilot systems are excluded from the main statistical claims. The preprint says the benchmark package—including prompts, ratings, audio and evaluation tools—is planned for release upon acceptance.