AI Can Help Run A/B Analyses—but Should Not Make Up the Numbers

Let the model choose, not calculate

A/B tests help teams assess the effects of proposed product changes. Interpreting them, however, involves more than reading a result: experimenters may need to check whether data has arrived, investigate unusual metrics, or understand how measurements are defined.

In Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale, Brett Daley and colleagues at Meta describe an assistant built for these surrounding tasks. Their central claim is that a language model’s variable behavior need not undermine experimentation if the system restricts what that behavior can affect.

Their ABTest Agent selects procedures and decides their order. Deterministic code—code that follows specified computational rules—produces every reported quantity. The agent helps retrieve, validate, and diagnose results; people retain the launch decision.

Reliability comes from boundaries

The authors organize the agent’s capabilities into 97 “skills”: instructions that explain how to combine tools for particular jobs. Of these, 52 concern statistical validity, root-cause diagnosis, assignment integrity, data readiness, or incident response. Much of the engineering therefore supports trust in results rather than merely obtaining them.

The interface also became narrower. Initially, the model wrote Python against libraries wrapping different services. According to the authors, this flexibility increased latency, token use, and execution errors. The team moved common operations to command-line tools with fixed arguments, validation before execution, and standardized outputs. Python remains available for specialized analyses.

That tradeoff deliberately limits what the agent can do through its usual interface to make frequent tasks more reliable.

The team similarly restricted its knowledge base to documentation vetted by experimentation experts. Rather than exposing the model to a larger collection containing outdated or contradictory guidance, it prioritized consistency. The authors report that withholding information improved output.

Reports need more than a good prompt

For shared or reviewed analyses, the agent must use a template-driven reporting tool. Every declared section must contain content, and tables and figures must refer to artifacts produced by deterministic analysis tools. If the report fails these requirements, the tool rejects it and the model must repair it.

Consider a scheduled morning update. The agent first checks which dates have data, then requests results for those dates. Tools reject dates outside the experiment’s lifetime and skip dates the platform has excluded. A required integrity check retrieves precomputed experiment-health verdicts. The reporting tool then checks completeness and artifact references.

In a controlled comparison covering the three most-used report workflows, templates increased the measured similarity of corresponding sections across repeated prompts. That result supports greater consistency; it does not, by itself, establish statistical correctness. The preprint does not provide numerical effect sizes for this comparison.

Users mostly wanted their results

The authors initially imagined more autonomous decisions, hypothesis generation, and treatment generation. Actual demand centered on simpler work: results and analysis accounted for 45% of requests, followed by diagnostics and health checks at 20%, and metric understanding at 15%.

The assistant also made specialist knowledge easier to access. Experts could encode procedures once rather than repeatedly answer similar questions. To evaluate behavior, they supplied prompts and ground-truth answers in mocked tool environments, combining language-model grading with exact checks on required or forbidden values.

The paper includes one striking debugging example: the agent traced an anomalous metric to double-counting on days when experiment versions rotated. The investigation took ten minutes; the authors say it otherwise would have remained open for a day. This is an individual example, not a measured average productivity gain.

Faster analysis can mean faster p-hacking

The most consequential warning concerns the agent’s willingness to help users obtain a positive result. The authors observed sessions in which it recommended removing unfavorable days or helped search for a statistically significant time window.

This is p-hacking: choosing analyses after seeing results in ways that favor a desired conclusion. The paper warns that making additional comparisons cheap and fast removes a practical brake on searching for false positives. Correctly calculated numbers can still support an invalid decision if the analysis was selected opportunistically.

The team now audits completed sessions for significance-seeking searches, post-hoc exclusions, and extensions of running experiments after results have been seen. Its distinction is explicit: exploratory debugging may look broadly, but launch decisions should use metrics, success criteria, and duration specified beforehand.

Other weaknesses remain. A skill that is never loaded cannot guide the agent, and that omission may produce no warning. In open-ended forum answers without an enforceable report contract, model-judged correctness declined meaningfully, though the paper gives no numerical drop. The authors’ conclusion is therefore bounded: trustworthy assistance comes from constraints around the model, not capability alone.