A multilingual test for AI ticket routing—and why its high scores need context

The question behind the benchmark

Many AI systems generate answers one word at a time. A typed-decision model instead scores a small set of predefined options in one pass. For a support desk, those options might be “billing,” “technical,” or “other.” That can make routing fast and predictable—but does it work when messages are written in languages beyond English, or mix languages in one sentence?

PolyRoute, a preprint by Mohammed Nayeem Khan and Sarun Kumar K S, creates a shared test for that question. It includes 322 support messages in 12 languages and nine writing systems, plus 156 code-mixed or romanised messages in Hinglish, Spanglish, and Arabizi. Each message is assigned to one of the three departments. The study focuses on this single, relatively straightforward task; it does not test multilingual AI in general.

How the test was made

The authors used a language model, Qwen3.8-27B, to generate messages for each department and language. They then asked the same model to label each message without seeing its intended answer, keeping items only when the labels matched. A different model, gemma4-31b, independently labelled the test messages as an additional check. It agreed with the supplied labels on 97.2% of the native-language items.

That agreement is reassuring as a consistency check, but it is not human verification: two models can share mistakes. No native speakers reviewed the messages, and both the text and labels were generated by AI. The authors also checked for near-duplicates between test and training texts; they found no exact duplicates, and the maximum reported character 5-gram similarity was 0.33. This reduces one kind of leakage concern, but cannot rule out every form of overlap or shortcut.

The comparison included released English and multilingual model checkpoints, versions fine-tuned on the task, and a conventional text classifier based on word and character patterns. The authors also tested variants trained with some native-language examples withheld. Importantly, the shared base training data already included generic multilingual examples, so “withheld” means reduced exposure—not zero exposure.

What the results show

On the 322 native-language messages, the English-root checkpoint scored 61.2% accuracy; the released multilingual checkpoint scored 77.3%. Fine-tuning the multilingual model on the task raised accuracy to 96.6%. A TF-IDF classifier—a method that learns from how often words and character patterns appear—reached 91.9% when trained on all available rows. That strong baseline matters: this routing task contains useful lexical cues, so the neural model’s score cannot be interpreted without comparison to simpler methods.

Fine-tuning also helped on the four languages designated as withheld: the multilingual checkpoint scored 76.7%, while the fine-tuned version without dedicated examples from those languages scored 90.3%. The paper cautions that these languages still appeared in the generic base data. Adding dedicated examples for all languages produced 93.2% on the four-language subset, but the difference from 90.3% was not statistically significant in this test.

Code-mixed messages tell a different story. The English-root checkpoint scored 84.6%, compared with 79.5% for the released multilingual checkpoint. The difference was not statistically significant (p=0.134), so the authors describe it as a direction in these results, not a reliable finding that English models are better at code-mixed text. Fine-tuned systems scored higher on this split, but there were only 156 items, and the authors say the two test splits measure different things and should be reported separately.

What remains uncertain

The benchmark’s central limitation is its synthetic data. Generated messages may be more regular than real customer writing, and the paper notes that dialects, typos, mixed scripts, and domain vocabulary are underrepresented. This is especially important for lower-resource languages, where no human review establishes that the examples sound natural. High accuracy here does not show how well a system serves actual speakers.

The test is also small. With 322 native-language items, the authors estimate that differences of roughly four percentage points or less are difficult to resolve for typical paired comparisons; each individual language has only around a few dozen messages. The top fine-tuned systems therefore cannot be confidently ordered by small score differences. The paper also finds that changing the option names can sharply reduce performance, underscoring that results depend on the exact question and schema.

PolyRoute is best read as a reproducible starting point for comparing models and identifying questions to investigate—not as a certification test for deployment. Its findings come from one routing task and a preprint, which may not have undergone peer review. The authors themselves call for evaluation on human-written messages from the communities a system is meant to serve.