Targeted AI repairs improve scientific code translation—but simple tests miss errors

Scientific software can compile successfully, finish without crashing and still perform the wrong calculation—or skip the intended work altogether. A new study of AI-assisted code translation finds that organizing repairs around specific failures improves results, while highlighting how easily weak tests can mistake broken programs for successful translations.

Jianuo Shen, Pengyi Xu and Bo Zhang describe the work in an arXiv preprint submitted on November 6, 2026. The paper is listed as accepted to AgenticAI4HPC 2026, an SC26 workshop. Its findings concern selected computational kernels, not complete scientific applications.

Dividing the work

The system translates among CUDA, OpenMP and SYCL, frameworks used to express parallel computation. Preserving a program’s meaning across them requires handling differences in memory management, synchronization and how work is assigned to computing devices.

Rather than repeatedly asking one model to fix whatever went wrong, the researchers divide the process among three software agents. The translation agent first summarizes the source program’s calculations, then plans memory use and generates the target code. A compilation agent invokes a real compiler and classifies errors. An execution agent runs the resulting program and examines its exit status and output.

A central controller routes failures to specialized repair routines. These address compilation problems, memory issues, numerical logic and synchronization. Revised code returns to compilation and testing. The experiments allowed up to nine repair attempts; programs that timed out could also be rerun with smaller numerical input arguments.

The distinction matters because not every failure needs the same response. A missing header and an incorrect reduction—a calculation combining results from parallel workers—call for different fixes.

Better results with fewer tokens

The evaluation used 30 kernels from HeCBench, covering areas including image processing, numerical methods, sorting and physics simulation. Six lacked OpenMP implementations, so directions starting from OpenMP used 24 kernels rather than 30.

The researchers tested six translation directions with three open-source language models: DeepSeek V4 Flash, GLM 5.2 and Kimi K2.7 Code. They compared their workflow with LASSI, an existing translation pipeline, using the same models, benchmarks and compilation and execution environments.

Under the initial checks, the new workflow achieved higher pass rates in all 18 model-and-direction combinations. Its per-model averages across directions ranged from 78.6% to 94.6%, compared with 34.5% to 65.5% for LASSI.

Average token consumption was 26.2% to 52.4% lower, depending on the model. Tokens are the chunks of text processed or generated by a language model; fewer tokens do not automatically establish equivalent savings in elapsed time, electricity or money. Nor were savings universal: the workflow used more tokens in two individual model-and-direction combinations.

Repair contributed substantially to the reported success. For OpenMP-to-SYCL translation, all three models initially passed no more than 12.5% of kernels, before reaching much higher rates after repairs. The authors report diminishing returns beyond roughly five rounds.

A successful run was not enough

The initial acceptance rule was notably limited: a program passed if it exited with code zero and printed no explicit FAIL marker. That is not a comprehensive comparison of numerical results with the source program.

To investigate missed defects, the researchers used DeepSeek V4 Flash as a separate, offline evaluator. It assessed cached artifacts using criteria intended to reject skipped workloads, numerical errors and unsupported success messages. This audit did not repair the programs.

Across both systems, the audit identified 117 false positives among 999 output-producing artifacts: translations that passed the initial checks but were subsequently rejected by the evaluator. The audited set also included initial failures; overall, 70.1% passed the initial checks and 58.4% passed the audit. The 117 figure therefore counts newly rejected apparent successes, not all audit rejections. Leading problems among these false positives included placeholder implementations, numerical errors and removed or skipped validation. Nine LASSI runs that produced no output were excluded from this audit.

The new workflow retained higher audited pass rates for each model: 71.4%, 68.5% and 86.3%, against LASSI’s 46.4%, 28.0% and 49.1%, respectively.

What remains unproven

The second check is itself a language-model judgment, not an independent mathematical guarantee or exhaustive numerical test. Its high self-reported confidence does not establish its accuracy, and using DeepSeek for both generation in some runs and evaluation limits independence.

The benchmark also does not demonstrate reliable translation of multi-file applications, preserved execution speed or universal GPU offloading. The artifact documentation supports CPU-only OpenMP execution, and performance-aware repair remains future work.

The study therefore supports a bounded conclusion: targeted repair improved translation outcomes on these kernels, but dependable scientific software still requires stronger validation than either a clean exit or an AI evaluator’s approval.