Adapting speech recognition to Nepali-English conversation takes more than changing scripts

For Nepali speakers who mix English into everyday conversation, an automatic transcript can fail in two ways: it can mishear a word, or recognise its sound but write it in an unexpected script. A new preprint examines both problems by adapting an existing speech model to transcripts that keep Nepali in Devanagari and English in Latin letters.

Jeevan Neupane and colleagues at AI Lab Pvt. Ltd., Nepal, report that adapting the 449-million-parameter SraVaani model reduced word error rate from 48.6% to 17.1% on their six-domain test set. Extending training from six to twelve passes through the training data brought the figure to 16.3%. However, performance on a separate read-speech benchmark deteriorated under its original transcription convention. The study is therefore as much about preparing and evaluating speech data as improving a model.

Making the transcripts fit

The researchers assembled 172.8 hours of speech spanning read sentences, engineering lectures, podcasts, news and talk programmes, conversations, and technical interviews. About 153 hours went into training, with the remainder reserved for validation and testing. Across the complete corpus, 43.8% of clips contained at least one English word.

For material collected from YouTube, the pipeline separated and cleaned speech, divided recordings at pauses, and generated initial transcripts with Gemini. Another language-model pass corrected the drafts, followed by human checks against the recordings.

A practical obstacle appeared in SraVaani’s tokenizer, the component that breaks written text into units the model can handle. Its existing vocabulary supported Nepali and English letters, but not digits and several common symbols. Nearly 30% of source clips had transcripts containing unsupported characters.

Rather than replace the vocabulary or throw those clips away, the team converted numbers and symbols into spoken words. That required more than a fixed substitution: a written number might have been spoken in Nepali or English, particularly in technical lectures.

Their rules generated alternative readings in both languages. The unadapted model then transcribed the audio, and the system selected a candidate by comparing approximate pronunciations rather than matching scripts directly. Ambiguous or poorly matching cases were rejected. Final labels also had to survive encoding and decoding without changes or unknown tokens.

Recognition gains beyond the alphabet

The main experiment updated the entire network, retaining its vocabulary and two output heads. Word error rate counts substitutions, deletions and insertions relative to the reference transcript; lower is better, but it is not simply the percentage of words recognised correctly.

The six-epoch model’s 17.1% test WER had a reported 95% confidence interval of 16.6–17.6%. A second training seed produced 17.0%. English-word WER fell from 80.4% before adaptation to 20.1%, while Nepali-word WER declined from 36.1% to 15.9%.

Those improvements were not merely a switch of alphabet. A pronunciation-oriented, romanised measure designed to reduce script penalties also improved substantially. Freezing the audio encoder and training only the remaining components produced a worse overall WER of 25.5%, suggesting that adapting the speech representation mattered alongside learning the desired written form.

Keeping numeric recordings mattered too. Excluding training clips whose original transcripts contained digits or symbols raised overall WER to 19.8%. On numeric test clips specifically, errors rose from 14.3% to 21.3%. This comparison also removes training hours, so it does not isolate label preparation from the benefit of retaining more relevant speech.

When the reference changes the result

The separate FLEURS evaluation exposed a tradeoff. Against its original Nepali references, SraVaani’s WER worsened from 22.6% before adaptation to 31.1% afterward. Those references often write English loans in Devanagari, unlike the training targets.

With alternative mixed-script references for the same audio, the direction reversed: WER improved from 30.0% to 23.4%. Yet an additional character-level measure ignoring script and spaces still worsened, indicating genuine recognition loss as well as a writing-convention mismatch.

The alternative references were generated with language-model assistance and automatically checked, not fully human-verified. The evaluation also excluded clips containing digits. It cannot establish robustness on unfamiliar spontaneous code-mixed speech.

Other limitations constrain the headline gains. Training and test splits were not guaranteed to use different speakers, potentially making in-domain results optimistic. Number labels selected using the base model may inherit its mistakes and still need independent listening checks. The best SraVaani result was statistically indistinguishable from a fine-tuned Whisper model’s 16.4% WER, and Whisper had a shorter, still-improving training run. The findings support careful adaptation and evaluation—not a general claim of superiority.