Can AI Models Help Check Each Other’s Explanations of Legacy Code?

The documentation problem

Modernizing old software requires understanding what it does, not merely translating its programming language. The preprint focuses on COBOL, used in longstanding business systems, where missing documentation and departing developers can leave important rules buried in code.

Large language models can generate readable explanations, but readability does not establish accuracy. Conventional evaluation often compares an AI summary with a human-written reference. That approach cannot directly help when the reference is precisely what an organization lacks.

Marco Colombo and colleagues at the University of Illinois Urbana-Champaign, Infosys and the University of Illinois Chicago propose combining several signals to estimate which summaries deserve closer inspection. Their goal is documentation triage, rather than a formal proof that an explanation is correct.

Three complementary checks

The proposed pipeline starts with code chunks and supplied descriptions of routines they call. These dependency descriptions are fixed and vetted. The method therefore avoids requiring a reference summary for each target chunk, but it still relies on contextual information and preprocessing supplied by an existing Infosys pipeline.

Three systems generate explanations of the same input: an Infosys OpenAI-based system, LLaMA 3.3-70B and Grok-3. A dedicated OpenAI evaluator compares each alternative with the Infosys explanation. It assesses meaning rather than wording, returning an agreement label, a numerical score, missing details, contradictions and a rationale.

The aggregation rule averages supportive scores, assigning zero support to a disagreement. It also specifies an additional penalty when both comparisons disagree. The intention is to avoid treating one dissenting comparison as decisive evidence against the anchor summary.

A second check examines perplexity, a measure derived from the probabilities a model assigns to its generated words or word fragments. Lower perplexity means the model found its own output more predictable. That indicates generation confidence, not necessarily correct understanding. The paper maps perplexity onto a zero-to-ten scale using thresholds calibrated on COBOL validation samples.

The third check asks an AI evaluator to inspect the code alongside its summary for omitted logic. Adapted from G-Eval, it considers branches, file operations, transformations and control flow. Three evaluations produce ratings from one to five, which are averaged and doubled. This measures documentation completeness, not how much code software tests execute.

In the full proposal, agreement and perplexity scores are combined, then multiplied by the coverage score divided by ten. Incomplete explanations consequently receive lower final scores even if their wording appears confident.

What the evaluation reports

The authors describe sampling 100 chunks from AWS CardDemo, an open-source COBOL credit-card application. Experienced mainframe specialists wrote reference summaries and assessed the generated explanations. References were thus used to validate the evaluation method, although the proposed scoring process does not require them during use.

The results table, however, contains only 30 chunks, despite its caption and surrounding discussion referring to 100. Of those 30, 17 received high confidence scores, and experts judged 16 acceptable. Ten received medium scores, with six acceptable and four needing corrections. All three low-scoring summaries were judged problematic.

The paper also reports eight summaries with omissions involving file-error handling, conditional branches or validation rules. Its percentage for these cases, 27%, likewise corresponds to approximately 30 examples rather than 100.

These counts suggest useful separation within the displayed sample. They do not establish that every problematic summary was caught: one high-confidence summary was unacceptable, and four medium-confidence summaries required corrections.

Why the evidence needs caution

The experimental procedure describes generating summaries with Claude Sonnet 3.5 and combining perplexity with coverage. It does not report the full three-generator agreement component in that procedure. The displayed findings therefore do not establish the performance of the complete ensemble advertised by the paper.

There are mathematical reporting issues too. As written, the agreement formula already produces zero support when both comparisons disagree, so subtracting another penalty and clamping the result at zero cannot lower it further. Some illustrative final scores also do not follow the stated perplexity normalization and coverage calculation.

The paper does not provide a completed quantitative baseline comparison or establish that its zero-to-ten scores represent probabilities of correctness. Its claim that reviewing only 43% of outputs would reduce manual effort follows from the displayed triage split, not a reported measurement of reviewer time or cost.

More fundamentally, models can share mistakes. Agreement cannot expose an interpretation that all generators and the evaluator misunderstand. The authors acknowledge this limitation, along with anchor bias and differences in summary detail.

Preprint status

The supplied arXiv record identifies version 1, submitted on 6 November 2026. It is a preprint; the record does not establish peer-reviewed publication. Its contribution is a proposed combination of checks and preliminary reported observations, with unresolved inconsistencies that limit conclusions about dependable operational use.