When an AI experiment’s frozen protocol needs a bug fix

Pre-registration records an experiment’s criteria and analysis plan before data collection, helping distinguish planned tests from decisions made after seeing results. Software creates a complication: a frozen plan can still contain an implementation that crashes, uses impossible parameters or quietly measures the wrong thing.

In a preprint originally posted on 3 November 2026, with the supplied original publication timestamp normalized to 2026-11-03T05:23:45.000Z, Peter Chen, Ni Zhang and Siyuan Zhang examine three repairs from their own research program. Their experiments used language models to propose executable rules for production scheduling, incentive design and task assignment. A programmatic verifier checked proposals against specified requirements, and held-out evaluation measured their performance.

The paper is not a demonstration that AI-generated rules outperform alternatives. Its subject is the experimental machinery surrounding those rules—and the evidence needed when that machinery changes.

Three repairs, three different risks

The scheduling experiment stopped after 37 pairs of instances had been saved. Its generator already redrew unsuitable training instances, but did not apply that behavior to held-out instances. When a held-out baseline could not produce a feasible schedule within the specified horizon, the run crashed.

According to the authors, the frozen protocol already required redrawing unsuitable instances in either group. The repair therefore brought the implementation into line with the written policy. They report checking the held-out baselines of all 37 saved pairs and retaining those data because the baselines were feasible.

That check has a deliberately narrow meaning. It supports the feasibility of the retained instances, not an unrestricted claim that every possible consequence of the repair was absent. Later generated instances were subject to feasibility-conditioned redrawing, as the protocol intended.

The incentive-design experiment exposed a different problem during a preliminary smoke test: its budget was incompatible with its participation requirements under the design’s payment constraints. The authors calculate a minimum total payment of 60 for 20 workers, exceeding the original budget of 50.

Before collecting experimental data, they raised the budget to 100. The same repair entry also added a participation constraint, exempted reference baselines from the verifier gate while retaining them in evaluation, and tightened the proposal prompt to reduce irrelevant parsing failures. These were substantive design changes, even though the authors classify them as leaving the numbered criteria, thresholds and scoring rule unchanged. The smoke-test output was discarded.

In task assignment, the evaluator modified a shared batch in place. Evaluating one mechanism therefore changed the state encountered by the next. The reported symptom was identical composite scores across mechanisms: plausible-looking output from a broken measurement process.

The repair made evaluation copy its inputs internally so repeated calls would not contaminate one another. Before data collection, the researchers also changed task counts and deadline ranges to seek a less degenerate coverage–fairness tradeoff. Again, smoke-test output was discarded.

A repair needs an evidential boundary

The authors organize patches along two axes: defect type and timing. Environmental defects concern pipeline boundaries, such as crashes; feasibility defects concern impossible or uninformative parameter regimes; correctness defects concern measurement semantics. A single patch can span more than one category.

Timing determines what must be explained about existing data. For a repair after collection begins, the authors propose a “verifiable non-impact claim”: provide the affected records and a check that readers can rerun, rather than simply declaring the data unaffected. The scheduling case is their only observed post-data example.

Their domain protocols prohibit post-data changes to judgment criteria, thresholds or scoring. The paper separately discloses a revision to a cross-domain analysis protocol, outside its three-case set. That boundary matters: the claim of unchanged domain criteria does not mean every analysis artifact remained unchanged.

A checklist, not a validated guarantee

The proposed nine-item checklist combines frozen criteria, append-only changelogs, pre-committed negative-result narratives, protections against held-out-data leakage, equal-evaluation-budget random controls, verbatim rejection logs, uncertain verdicts for unevaluable properties, discarded smoke outputs and explicit patch classification.

All nine practices appeared in at least one case. That is evidence of use, not evidence that the checklist is sufficient to prevent misleading research.

The limitations are substantial: three cases, one research program, classification by its own authors and simulation studies rather than human-subject experiments. Checks can themselves contain bugs. The new budgets, task counts and deadlines were also selected through author judgment, with rationales recorded afterward—not through a pre-committed selection rule.

The useful distinction is therefore between making repairs inspectable and proving them harmless. This preprint offers a framework for the former; its small self-audit does not establish the latter.