A speech recognizer built to fit in your browser

Speech recognition often relies on large neural networks, and supporting several languages can mean storing several separate models. That creates a practical hurdle for phones and browsers, where download size, memory and computing speed matter. In a new arXiv preprint, Hemanth HM presents Swanada, a system intended to recognize speech locally across twelve languages and scripts, including Classical Sanskrit and Indian English.

The paper tackles two connected questions: Can a multilingual recognizer be made small enough for constrained devices, and can it produce useful text as someone speaks without revising words it has already finalized?

One shared model, separate language outputs

Swanada processes audio at 16 kHz. At its core is a Conformer, a neural network architecture that combines attention and convolution to interpret speech over time. Rather than deploy a complete network for each language, the system shares the speech-processing component and attaches one small output layer—or “head”—per language. Each head maps the shared representation to characters or subword units in a particular writing system. The runtime can switch heads without reloading the main model.

The authors describe two sizes. The Studio model is reported as 177.38 MiB; the Mobile model, a smaller version trained using the larger model as a guide, is 25.35 MiB. The paper also describes software adjustments intended to keep browser exports compact and compatible with WebAssembly, the technology that lets code run in browsers. For streaming, Swanada keeps a rolling audio window and separates text into a finalized portion and a changeable tail. Its rule is that finalized text should only grow, not be rewritten.

What the evaluations report

The authors tested the system on 480 held-out utterances, plus 1,280 public test clips drawn from sixteen benchmark sets. They report character error rate (CER), which measures character differences between a transcription and its reference; lower is better. On the 480-utterance set, the Studio model’s average CER was 5.43%. Across twelve languages in an open benchmark subset, it was 8.87%; across all sixteen open test sets, it was 12.06%.

The Mobile model’s CER was higher: 18.56% on the held-out set, 19.23% on the twelve-language open subset and 21.85% across all sixteen sets. The authors also report a median time to first token of about 31 milliseconds and no finalized-text revisions in 72 streaming sessions. These figures describe the tested setup, not a guarantee for every phone or recording condition.

The paper reports that the split design reduces storage for a twelve-language collection compared with keeping twelve separate large models. Its figures are 183.4 MiB for the Studio collection and 31.39 MiB for Mobile; the first Mobile load is reported as 25.85 MiB. The system also includes audio resampling and a fallback for certain mobile recording interruptions. These are engineering features, not evidence that every browser handles audio identically.

Where performance falls short

Results vary substantially with the speech. The paper reports high error rates for spontaneous, overlapping meeting speech and reverberant Sanskrit recordings, particularly on Mobile. The authors also note that character error can understate problems with whole-word accuracy in languages where a long word may be counted as wrong after a small spelling difference. Swanada requires the user to select a language, or a separate language-identification step; its script-specific heads do not automatically determine what is being spoken.

The evaluation is broad but bounded: the authors report 40 held-out studio utterances per language, selected samples from public test sets and 72 streaming sessions. Those results cannot establish performance across all accents, devices, noisy environments or everyday conversations. The source is an arXiv preprint, which may not have undergone peer review. Its measurements and design claims should therefore be treated as the authors’ reported findings, not independent confirmation. Running recognition locally may reduce the need to transmit audio, but actual privacy depends on how an application is built and used.