Can AI find the right peer reviewers?

Matching a manuscript with suitable peer reviewers is a practical challenge: editors must find people with relevant subject knowledge, while avoiding candidates whose apparent fit is too broad or superficial. Recommendation software can help sort a large pool, but judging it is difficult. Past assignments provide useful examples, yet they are sparse: researchers who were never selected may still have the right expertise. A system evaluated only on reproducing past choices could mistake “not previously assigned” for “not qualified.”

In OmniReview, Yehua Huang and colleagues build a dataset and evaluation framework intended to address that problem. They link records from the open-access publisher Frontiers with researcher and publication information from Open Academic Graph and ORCID. The paper reports 202,756 review records and 2,287,697 candidate reviewers, including 150,287 historical reviewers. The dataset’s released records use identifiers to point back to external metadata rather than redistributing all of it.

The team first matches publications across sources using normalized titles, then links researcher identities using publication overlaps and name information. They report manually auditing five samples of matched records, with 100 pairs checked in each round. Research topics are organized in a hierarchy: publication titles and abstracts are assigned to categories using an embedding model, and a scholar’s profile is built from those assignments. In a separate check of topic assignments, the authors report a mean relevance rate of 78.44% across five verification rounds. That figure is a reminder that automated categories are useful approximations, not ground truth for every publication or person.

The benchmark tests four different abilities. Can a system give high suitability scores to people who actually reviewed a paper? Can it suppress candidates who seem broadly related but lack specific expertise? Can it rank historical reviewers above other qualified candidates, and qualified-but-unassigned candidates above unqualified ones? The last task directly acknowledges that the historical record does not list every suitable reviewer.

The proposed model, Pro-MMoE, represents a candidate’s research in two ways. One branch uses summaries generated by a large language model to capture a publication’s central question, methods and findings; the other retains original titles and abstracts. A task-adaptive mixture-of-experts component then routes information toward separate objectives for suitability scores and rankings. The authors argue that combining summaries with source text can preserve broad meaning while retaining details that distinguish closely related areas of expertise.

In the reported comparison, Pro-MMoE achieves the best score on five of six metrics. Its clearest reported gain over the strongest competing method, CoF, is on suppression: the average suitability score assigned to unqualified candidates is lower, while the score for historical reviewers is slightly higher. On ranking qualified candidates, however, CoF has the better mean average precision; Pro-MMoE leads on the other reported qualification ranking metric, NDCG. So the model does not win every task, and different measures favor different aspects of recommendation.

The authors’ ablation tests—experiments that remove or change parts of the system—suggest that both summary and raw-text branches contribute to performance. Replacing publication-level summaries with one summary for an entire reviewer profile produced a particularly cautionary result: endorsement scores rose, but scores for unqualified candidates also rose sharply and ranking performance worsened. A high score for known reviewers, by itself, is not evidence that a system can distinguish good matches from poor ones.

These findings are about performance on a constructed benchmark, not evidence that the system improves editorial decisions in practice. The data come from Frontiers, so results may not transfer to other publishers, disciplines or review workflows. Publication records and historical assignments are proxies for expertise and editorial judgment; they do not establish availability, willingness, conflicts of interest or the fairness of past choices. The authors also note that their main comparisons are single-run results without variance estimates or significance tests, making small differences uncertain. Summaries can omit details or introduce unsupported interpretations, and generating summaries and embeddings requires substantial preprocessing, even though scoring with cached representations is much faster.

OmniReview offers a broader way to test reviewer recommendation than simply replaying old assignments. But a benchmark score cannot decide who is suitable for a real manuscript. The authors frame the tool as support for editors, and warn that treating its scores as final decisions could weaken human judgment or lead to unfair assignments. The paper is an arXiv preprint; its claims should be read as reported research findings, not as settled evidence of real-world effectiveness.