Specialist AI assistants do not always give better regulatory answers

Testing expertise rather than polish

Can a purpose-built regulatory assistant answer legal questions better than a general-purpose system? In a November 2026 preprint, Marco Virgolin, Thomas Haszprunar and Maximilian Regenold investigate that question through the EU AI Act Q&A Benchmark and an associated challenge.

Their central finding is selective rather than sweeping: two specialist entrants outperformed every baseline in both evaluation modes, while three did not. The results support the authors’ argument for testing regulatory assistants across several dimensions before considering adoption—not treating specialization or fluent language as evidence of reliability.

What the benchmark measures

The benchmark contains 110 open-ended questions, ranging from definitions to scenarios involving several actors and obligations. Each item includes an exemplary answer, explicit correctness criteria and expected legal references. One author performed the annotations, which a second author reviewed. Limited AI assistance helped discover possible questions and refine the evaluation criteria.

The legal scope is deliberately fixed: Regulation (EU) 2024/1689, interpreted against the state of affairs on 1 May 2026. Subsequent legal developments fall outside the evaluation. The scores therefore concern that defined snapshot, not necessarily the framework applicable when someone reads the paper.

Eight metrics separate answer correctness, citation quality, conciseness, professional tone and response speed. “Loose” answer correctness measures the proportion of individual criteria satisfied. “Strict” correctness measures the proportion of questions for which every criterion passes. Citation scoring likewise distinguishes finding the right Article or Annex from identifying the expected sublevel.

The overall score is the geometric mean of those eight metrics. It is not a percentage of legally correct answers: verbosity, extra references and latency also affect it. The authors chose this combination to penalize systems with a serious weakness in any measured dimension.

Who performed best?

Five purpose-built assistants participated: four commercial entrants and one university lab. They were compared with three baselines: GPT-5.5 with web search, GPT-5.5 with basic retrieval-augmented generation, and an unnamed search-integrated model. Retrieval-augmented generation supplies relevant document excerpts to the model before it answers.

Participants did not receive the benchmark questions in advance. Approximately 50 entities were contacted, and five agreed to participate.

Pharos Labs led the overall rankings, scoring 91.65% in single-question “easy” mode and 91.50% in “hard” mode. Ansvar AI followed at 86.64% and 86.29%. These were the only specialist entrants to beat all three baselines in both modes. The retrieval baseline placed third, ahead of the GPT-5.5 web-search baseline.

Those composite rankings conceal important differences. In easy mode, Pharos Labs passed every correctness criterion on 89.1% of questions. Ansvar AI’s corresponding score was 90.3%, despite its lower overall ranking. Pharos distinguished itself through balanced performance, including concise answers and relatively few excess citations.

Sounding professional was not enough

Professional tone barely separated the systems: 99.2% of tone judgments passed across the evaluation. Yet the best and worst overall scores differed by roughly 30 percentage points.

Citation precision exposed another gap. Every system scored lower when references had to identify the expected sublevel rather than merely the correct Article or Annex. The authors also report only a weak association between answer correctness and reference correctness. A broadly correct explanation did not reliably come with the expected supporting provisions.

Reported failures included missing deployer obligations in a contractor scenario and omitting data-protection compliance from an answer about sensitive personal data. However, some failures reflected stricter editorial judgments: the benchmark rejected “not necessarily” where its criteria demanded an unambiguous negative answer. The authors acknowledge that such cases may partly reflect annotator preference.

Pushback sometimes helped

Hard mode added a fixed synthetic conversation history and then challenged the assistant’s initial answer as potentially containing hallucinations. Only its revised answer was scored.

This did not consistently make performance worse. The leading systems changed little, while the unnamed search-integrated baseline improved its overall score by 5.06 percentage points. The authors suggest that challenging a weak initial answer sometimes functioned as an opportunity to improve it, rather than inducing an incorrect concession.

What the results cannot establish

Correctness and tone were judged using GPT-5.4, with three repeated evaluations. A human reviewer agreed with all 50 sampled judgments, although the paper explains that sampling different decisions in two inconsistent cases would have reduced agreement to 96%.

The authors acknowledge possible single-model judging bias and limitations from the small annotation team. More importantly, satisfying the checklist does not guarantee that additional claims are true: hallucinations were not directly evaluated, nor was support from each system’s retrieved material.

Funded entirely by regenold GmbH, the study releases its benchmark and evaluation code. Its conclusion remains conditional: strong performance here is promising, but readiness for broader regulatory work requires sustained performance across larger bodies of regulation, other workflows and further deployment-oriented validation.