Stricter AI checks did not consistently improve research results in two tests

The study separates two jobs often bundled together in automated research: proposing candidate rules and deciding which ones survive. Its central question is whether stronger screening improves the performance of the selected rule on previously unused data. Here, “discovery” means finding useful decision rules within restricted mathematical families, not producing and validating a broad scientific breakthrough.

The researchers studied job-shop scheduling using instances derived from production orders, and simulated vehicle-computing offloading using Beijing T-Drive taxi trajectories. Offloading rules determine how much computing work vehicles send to roadside infrastructure. Neither experiment involved deploying the discovered rules on real infrastructure.

Each task had 100 matched pairs of sessions. A small language model, Qwen2.5-0.5B-Instruct, generated 20 proposals with feedback from the standard verifier. A matched control supplied 20 random proposals from the same rule family. Three screening tiers then independently filtered these fixed pools. This design isolates changes in selection: it does not test how proposal generation would change if the model received feedback from each different tier.

The tiers ranged from minimal checks to fuller constraints and an additional stress test. In the vehicle task, the strict tier tested every task with channel quality reduced to the minimum for that instance. Each tier selected its best passing candidate according to training performance, then evaluated that rule on held-out data. The main outcome was the selected model proposal’s score minus the selected random proposal’s score—not simply the model rule’s absolute performance.

Before the main experiment, the authors tested whether stronger verifiers actually rejected more deliberately varied candidates. Across a 429-case mutation battery, rejection rates increased with strictness in both domains. However, separation was weak in the vehicle task: adjacent tiers differed by only two rejected candidates out of 310, with overlapping uncertainty intervals. The checks therefore met the registered ordering criterion, but their practical differences were not equally strong in both tasks.

The pre-registered prediction required the model-over-random advantage to increase strictly through all three tiers. That pattern was not established. In scheduling, the strict-versus-weak comparison had a one-sided p-value of 0.1115, above the registered 0.05 threshold. This does not prove that stronger verification has no benefit; a modest improvement remains compatible with the evidence. Two scheduling pairs also involved extreme failure penalties. Removing them changed the average margins but left that test’s p-value unchanged.

The vehicle result pointed in the opposite direction to the prediction. Across 97 pairs valid for the three-tier comparison, the model’s average advantage fell from about 0.102 under weak screening to 0.062 under strict screening. An exploratory analysis estimated the reduction at 0.040, with a 95% bootstrap interval from −0.07 to −0.01. The model still outperformed random proposals at every tier in that domain. A smaller relative advantage should not be confused with a universal decline in absolute quality.

There was also a structural problem with the registered prediction: the weak and standard gates selected identical model candidates in all 100 scheduling pairs and 99 vehicle pairs. Where selections coincide, their performance cannot be strictly ordered. The authors acknowledge this as a lesson in designing reachable pre-registration criteria. Extra checks can exist without changing which candidate wins.

An exploratory analysis asked what would happen if both proposal pools were merged. Selection still used training scores, not advance knowledge of held-out results. In scheduling, combining 20 model proposals with 20 random proposals increased held-out value by a reported 1.52 task-specific units over the model-only pool, with a one-sided p-value of 0.0002. But the combined pool contained twice as many candidates. This was not a pre-registered, equal-budget experiment isolating diversity from pool size. The vehicle task showed no corresponding improvement; estimates were negative and did not meet a 0.05 significance threshold.

Another post-hoc analysis suggested that strict screening’s usefulness depended on operating conditions. Re-evaluating selected vehicle rules under different loads produced a negative strict-versus-weak contrast at low utilization and positive contrasts at higher utilization, peaking at intermediate loads. Task density changed alongside utilization, however, so the analysis cannot cleanly identify utilization as the cause.

The scope remains narrow: one small model, two rule families, random proposals as the only baseline, and simulations or resampled records rather than deployment trials. Scheduling instances reused a small order pool, limiting independence and generalization. The study does not compare absolute quality against human-designed or classical heuristics. Its strongest message is conditional: stronger filters did not reliably increase the model’s relative advantage in these pipelines. Pool composition is a promising hypothesis for follow-up experiments, not an established replacement for verification.