Language models help design rewards for two-stage payment fraud detection

Payment fraud screening is not necessarily a single decision. A platform can assess a purchase before the card issuer checks it, then reassess it after authorization. Decisions at the first checkpoint affect what reaches the second. Blocking too aggressively can reject legitimate business; blocking too cautiously can let fraud proceed further through the payment process.

A September 2025 preprint by Bo Qu and colleagues investigates whether large language models can help train a system to coordinate those decisions. The language model does not directly approve or reject purchases. Instead, it writes reward functions: rules that assign rewards and penalties to the decisions made during reinforcement-learning training.

Turning payment screening into a sequence

The researchers model the platform’s pre-authorization and post-authorization checkpoints as a two-step decision process, with issuer checks between them. At each platform checkpoint, an agent can block or allow a transaction. Its inputs primarily use scores from existing supervised fraud models, together with information about the transaction stage.

The agent is trained offline on historical transactions using REINFORCE, a policy-gradient algorithm. This means the reported experiments learn from recorded data, rather than exploring new blocking strategies on live customers.

The business objective is expressed in transaction dollars: catch fraudulent value while limiting legitimate value incorrectly blocked. The design also favors catching more fraudulent value at the earlier checkpoint. The authors first construct human-designed rewards using stage-specific precision requirements, then ask whether language models can improve on them.

A language model writes the training incentives

The framework supplies a language model with the task description, training-code context, definitions of fraud outcomes and requirements for executable reward code. Candidate functions undergo basic structural checks before being used to train an agent.

The resulting agent is evaluated, and performance feedback is returned to the language model. Better functions become examples for subsequent attempts; unsuccessful attempts can prompt a written reflection that informs the next round. This is an iterative search over training incentives, not evidence that the deployed system continually rewrites itself.

The study tests Mixtral-8x7B, LLaMa-3-8B and Gemma7B. In the authors’ “few-shot” setup, prompts include human-written reward examples. In the “zero-shot” setup, they omit those examples but still provide domain descriptions, objectives and feedback. Zero-shot therefore does not mean learning fraud detection without transaction data or task guidance.

What improved in the tests

Training used approximately 2.14 million transactions from September 1–14, 2023. A short-term evaluation covered 522,105 transactions from September 15–30. A longer evaluation included approximately 6.17 million transactions from November 2023 through April 2024.

The main measure was dollar-weighted precision at specified dollar-weighted recall levels. Recall measures the share of labeled fraudulent transaction value caught. Precision measures the share of blocked transaction value labeled fraudulent. At matched recall, higher precision means less legitimate value is blocked relative to the fraudulent value caught.

On the short-term dataset, at 85% dollar-weighted recall, the pre-authorization baseline achieved 58.79% precision. The agent trained with a human-designed reward reached 64.22%. With LLaMa-3-8B-generated rewards in the few-shot setting, precision reached 71.70%—12.91 percentage points above the baseline and 7.48 points above the human-reward agent.

Results varied by language model, prompting setup and recall target. At 90% recall, for example, few-shot Mixtral reached 58.00% precision, compared with 51.27% for the baseline. Few-shot prompting was not universally better than zero-shot prompting across every reported comparison.

The authors also report that the agents retained an advantage over the baseline across the six-month evaluation. They report production inference latency below 50 milliseconds on standard CPUs for the compact policy network; this is not the time required for language-model inference or reward optimization.

What the evidence leaves open

The baseline matters. It uses a pre-authorization supervised-model score, rather than a separately optimized supervised policy spanning both checkpoints. The results therefore do not establish how much improvement comes specifically from reinforcement learning versus coordinating decisions across stages.

The reward search also uses evaluation feedback to select candidates. Consequently, the distinction between data used for reward selection and independently untouched evaluation is important when interpreting the gains. The reported tables do not provide confidence intervals for those differences.

Finally, six months of historical evaluation cannot establish resilience to every future fraud strategy. The authors acknowledge that some generated reward choices remain difficult to interpret and identify online reinforcement learning as future work. The preprint supports language-model-assisted reward design as a promising approach in this payment setting, not a general guarantee of autonomous fraud prevention.