Expert Feedback Helps AI Refine Finance Briefs—but Strong Answers Can Get Worse

For specialized writing, a useful AI answer needs more than fluent prose. Financial business briefs must balance relevant detail, clear organization, appropriate sourcing and professional expectations. Creating reference answers that capture those qualities can require considerable expert time.

In an arXiv preprint, Microsoft researchers Honson Ling, Kidus Admassu and Stephanie Rosenthal investigate whether experts can contribute through short critiques instead of drafting complete answers. Their workflow, Subject Matter Expert Critique, or SME-C, combines professional judgments with language-model analysis and revision. The finance case study reports improved average quality ratings—but also shows why automatic rewriting should not be applied indiscriminately.

Experts judge; models revise

SME-C starts with domain-relevant queries and several AI-generated answers to each. Professionals compare the candidates, choose their preferred responses and explain strengths and possible improvements. Researchers or language models then analyze that feedback to identify recurring expectations and query-specific revision needs.

The final stage produces evaluation rubrics and revised examples. The rubric captures criteria for assessing future answers; the examples illustrate responses shaped by professional preferences. Revised answers can subsequently enter another evaluation round.

The researchers tested two revision strategies. Critique-based refinement used insights distilled from professionals’ written comments. Vote-based refinement used the human-selected winner and alternative answers, asking the model to infer strengths, gaps and relevant domain practices without the written critiques. The latter reduces the feedback required from people but places more responsibility on the model’s own understanding of finance.

A finance case study

The initial collection covered 520 queries about publicly traded companies, spanning subjects including financial performance, regulatory issues, competitive positioning and technology. Three anonymized, web-enabled chat models generated candidate answers, with their original formatting and citations preserved.

Across a pilot and a larger collection round, 411 finance and business professionals contributed 3,853 evaluations, described in the paper as critiques. Participants typically evaluated ten queries in approximately 20–30 minutes and received $15. Their expertise was relevant but not necessarily deeply specialized for every question.

The feedback emphasized both substance and presentation. In the pilot, depth of information appeared in 38% of evaluations as a valued quality, while clarity and brevity appeared in 30%. Common improvement requests included better headings, summaries or tables, and citations. These findings informed rubric dimensions covering depth, clarity, organization, summaries and comparisons, and sourcing.

Both revision methods were applied to all 1,257 candidate answers from the larger collection round. Validation, however, examined only a selected subset.

Modest average gains, different strengths

For the blind validation, the researchers selected the 50 queries with the highest agreement about the original best answer. One hundred additional professionals compared that winning answer with its critique-refined and vote-refined versions. Version labels were randomized, and each participant evaluated five queries, producing 500 evaluations.

Mean quality ratings on a ten-point scale rose from 6.87 for the original answers to 7.33 for critique-based revisions and 7.29 for vote-based revisions. The paper reports statistically significant improvements over the original for both methods, but no significant difference between the two revision strategies. Median ratings increased from seven to eight.

Qualitative analysis suggested different strengths. Critique-guided revisions offered more contextualized detail and domain-grounded organization. Vote-guided revisions sometimes removed irrelevant material more effectively. These patterns do not establish that either method was superior overall.

Nor did revision reliably win on every query: the original answer had the highest mean rating on 18 of the 50 queries. The largest gains occurred among weaker baselines. Conversely, originals rated nine or ten declined by an average of 2.2 points after refinement, which the researchers attribute to unnecessary restructuring that weakened already-specific answers. They therefore argue for selectively revising weaker responses rather than repeatedly rewriting everything.

Better references are not verified facts

The study’s design limits what its quality gains establish. Side-by-side comparisons may emphasize visual structure over close reading. Written feedback was sometimes sparse, and participants’ expertise varied across topics. Selecting only high-consensus queries also leaves uncertainty about performance where professionals initially disagreed.

Most importantly, refinement can introduce hallucinations. The authors position SME-C primarily as a way to capture communicative effectiveness and professional utility, not as a substitute for additional factual or numerical verification.

The proposed efficiency advantage also needs careful framing. The paper estimates that expert authoring takes two to four hours per response, compared with less than five minutes for an individual critique, but does not report a controlled head-to-head authoring comparison. Its evidence supports a promising division of labor for curating evaluation resources—not proof that polished reference answers are financially correct or that the same gains extend beyond finance.