Confucius4-R2T2 adapts speech recognition to release text it will not revise

Speech recognition systems often revise their early guesses as more audio arrives. That can be tolerable for captions, but troublesome when another machine immediately translates the words or acts on them. Confucius4-R2T2 aims to make those incremental transcripts dependable without giving up much recognition accuracy.

In a preprint submitted to arXiv on November 4, 2026, Daniel Huang and colleagues at NetEase Youdao and UC San Diego describe adapting Qwen3-ASR-1.7B into a streaming recognizer whose released text is append-only. Their experiments show substantial improvements over the unadapted model at short audio intervals. The results concern recognition accuracy and transcript emission timing, however—not measured end-to-end deployment latency.

Learning what to say—and when to wait

The central training idea is the authors’ formulation of a Longest Stable Prefix, or LSP. They run the original recognizer on progressively longer audio clips and compare its outputs with its own transcript of the complete recording. At each point, the target is the longest whole output already produced that exactly matches the beginning of that full-audio transcript.

Requiring a whole earlier output matters. A partial hypothesis with a correct beginning but an incorrect final word does not earn credit for that beginning through this particular hypothesis. The rule is deliberately conservative and can delay emission compared with targets that retain any matching initial portion.

These labels are constructed offline, with the complete recording available. The deployed model does not see future audio; it learns from those retrospective examples when the available context warrants another commitment. Training also introduces a Wait token for uncertain continuations.

The authors mathematically establish that their labels never shrink, agree with the teacher’s full-audio output, and are longest within the restricted class of matching whole earlier outputs. Those properties do not prove that the words are correct: the anchor is a model-generated transcript, not human ground truth. Nor do label properties alone guarantee a trained network’s behavior. R2T2’s decoding policy preserves already committed text.

Adapting the decoder, not rebuilding the recognizer

The team post-trained the model on approximately 1,500 hours of proprietary Chinese and English audio. They froze its audio encoder and projector, updating only the language decoder through supervised fine-tuning followed by reinforcement learning. No additional architectural layers or prediction heads were introduced.

At inference, users can adjust the audio chunk size and optionally withhold a small provisional suffix. In the main quality-oriented setting, one token remains internal and revisable; released text still cannot change. A lower-lag setting withholds no tokens. Caching and bounded audio context reduce repeated computation, although the active audio window and retained transcript still require processing at each update.

Close to full-context accuracy on the main tests

Across eight English and five Chinese test sets, R2T2 with 160-millisecond chunks and one withheld token achieved a combined macro error of 5.76%, compared with 21.01% for the base model at the same chunk size and token budget. The base model’s full-context result was 5.15%.

That combined measure averages English word-error percentages and Chinese character-error percentages equally across datasets. It is not a pooled error rate over all words and characters. R2T2’s streaming result was about 0.61 percentage points above the full-context reference.

R2T2 also had the lowest combined macro error among the streaming configurations ranked in the main comparison, including three proprietary APIs. That does not mean it won every individual benchmark: two commercial systems had better Chinese macro accuracy, alongside later transcript stabilization.

The adaptation transferred beyond its two post-training languages. On 18 test sets covering ten additional languages, R2T2 improved over the base model at matched 160-millisecond settings on every test set. Its average error was 8.40%, versus 30.75% for that streaming baseline, but remained above the base model’s 5.99% full-context result.

What the timing results leave open

At 80-millisecond chunks without a provisional suffix, the authors report mean retrospective emission lag below half a second in both English and Chinese. This measures audio-timeline delay relative to aligned reference speech, excluding execution, queueing, transport and rendering. The study did not verify that inference consistently keeps pace with incoming audio.

Lag scoring also includes only correctly matched words or characters and excludes high-error utterances. It must therefore be read alongside recognition error, not as a complete responsiveness measure.

Further limitations include proprietary post-training data, possible evaluation-data exposure during the base model’s pretraining, and missing controlled ablations needed to isolate individual reinforcement-learning reward effects. The authors report releasing code and model weights. The preprint offers evidence for stable incremental recognition, while real-world serving performance and broader generalization remain to be established.