AI claims-analysis preprint tests policy citations and privacy safeguards on synthetic data

How the system works

The framework divides claim analysis among four specialist agents, coordinated by the gpt-oss-120b language model. One reports denial risk, another explains the prediction, a third examines combinations of diagnosis, procedure and medication codes, and a fourth explores hypothetical changes to a claim. A separate privacy reviewer checks the resulting report.

The predictive component uses XGBoost, a machine-learning method built from decision trees. Its probabilities are calibrated to make risk estimates more interpretable. An explanation tool identifies features pushing a prediction toward denial or payment, while graph-based models examine relationships among clinical codes.

The hypothetical-change tool recalculates risk after modifying features such as authorization status or processing time. These are model-based what-if scenarios: they show how predictions respond to changed inputs, not whether an intervention would cause an insurer to pay.

For policy support, the system uses retrieval-augmented generation, or RAG. It searches a document collection using both exact-word matching and semantic similarity, then supplies selected passages for the report to cite. The collection covers topics including eligibility, inpatient services, hospice care and appeals. Its payer policies were adapted from public sources and labeled fictional.

What the researchers tested

The dataset contained 99,079 eligible synthetic claims, with 27.4% classified as denied. That category included both rejected and partially paid claims. The researchers used 79,263 claims for training and 19,816 for testing.

On the held-out test set, the prediction model achieved a ROC-AUC of 0.90, a measure of its ability to rank denied claims above paid ones. Precision was 0.80 and recall was 0.76. These results describe performance on generated data, not hospital billing records.

The full workflow was evaluated on 100 held-out claims already classified as high risk: half had predicted denial probabilities between 50% and 80%, and half above 80%. Each received the same neutral question without instructions naming particular tools.

For every run, researchers retained the coordinator’s draft and compared it with the privacy-reviewed version. Both versions used policy retrieval, so this comparison isolated the downstream privacy layer rather than testing whether RAG improved performance over a system without retrieval.

Traceable numbers and retrieved citations

The coordinator selected all four analytical agents and policy retrieval for every evaluated claim. An independent gate checked for missing steps but did not need to supply any.

Reports averaged 98% numerical agreement with the evidence returned by tools and retrieved passages. This measures whether numbers were traceable to supplied evidence; it is not a finding that 98% of all statements or recommendations were correct. The paper did not calculate a confidence interval for this report-level average.

Of 216 policy citations, 214 referred to passages actually retrieved for the question. That verifies citation provenance, but does not by itself establish that a passage supports every conclusion drawn from it. In the illustrated case, the system lacked a payer-specific authorization rule and relied partly on general guidance.

Among the 100 selected high-risk claims, 81 were actually denied within the synthetic dataset. Hypothetical changes reversed the model’s predicted decision for 63 claims, most often by changing processing time. Neither result demonstrates that the proposed actions would prevent real denials.

Privacy gains, with boundaries

Draft reports contained 280 detected identifier occurrences: 138 payer names, 141 raw claim identifiers and one protected-health-information pattern. After review, none remained detectable by the scanners. The framework replaced payer names and claim identifiers with stable pseudonyms, while preserving the analytical content, according to the authors.

The language-model reviewer marked 98 of 100 reports compliant across its HIPAA, GDPR and payer-anonymity checks. Those judgments are not legal certification. The authors explicitly state that the compliance agent cannot establish legal compliance or replace human oversight. Cloud deployment would also require appropriate contractual, security and data-protection arrangements.

What remains unproven

The evidence comes from one synthetic dataset and a small, deliberately high-risk sample. It does not establish performance across ordinary claim workloads or real patients’ records. Detecting no remaining identifiers under the chosen scanners is also narrower than proving that every possible disclosure risk has been eliminated.

Accessibility remained uneven: only 23 reports avoided all flagged technical jargon. Analyses averaged 17 seconds, but the study did not measure fewer denials, recovered revenue, staff productivity or operational savings.

The contribution is an evaluated way to assemble analytical tools, policy references and output privacy checks. Whether that combination improves healthcare billing in practice remains a question for larger, real-world validation with human decision-makers.