When machine learning meets weak instruments, precision can mislead

The problem behind a precise-looking estimate

Economists use instrumental variables to study relationships that ordinary comparisons may misrepresent. An instrument is a variable associated with a treatment that, under the required assumptions, affects the outcome only through that treatment. The paper’s examples include using lottery selection to study Medicaid coverage and college proximity to study schooling.

An instrument becomes weak when its relationship with treatment, after accounting for background characteristics, provides little identifying information. In that setting, familiar confidence intervals based on an estimate and its standard error can be unreliable. A near-zero estimate with a small reported standard error does not necessarily establish a precisely measured zero effect.

Meng’s arXiv preprint investigates this problem in double machine learning, or DML. No journal publication is listed in the supplied source.

What machine learning removes—and what remains

The study uses a partially linear instrumental-variable model. It represents the treatment coefficient as constant while allowing background characteristics to relate flexibly to the outcome. Machine-learning models estimate the predictable components of the outcome, treatment and instrument. Subtracting those predictions produces residuals: the variation left after accounting for the controls.

These prediction models are called nuisance functions because they support estimation rather than being the main quantities of interest. Their quality matters, however, because errors in removing predictable variation can affect both instrument strength and inference.

The procedure separates model fitting, model selection and final evaluation. Models are fitted on one subsample, selected on another and evaluated on a held-out fold. The selected models are not refitted on the combined fitting and selection samples. For clustered observations, entire clusters remain together throughout these splits, preventing dependent observations from crossing the training–evaluation boundary.

Testing effects without assuming a strong instrument

The main inferential tool is the Anderson–Rubin test. Rather than relying on the estimated treatment coefficient’s usual normal approximation, it tests whether each proposed coefficient is compatible with the residualized data. Collecting the values that are not rejected produces a confidence set.

Building on an existing independent-observation baseline, Meng establishes large-sample validity for clustered, cross-fitted Anderson–Rubin inference under stated sampling and nuisance-estimation conditions. The guarantees allow instrument strength to approach zero. A complementary score test also receives a validity result; with one instrument, it coincides with Anderson–Rubin inference.

Extensions address survey weights, which can change the target population and the conditional means being estimated, and instruments constructed from data. Constructed instruments must exclude their own inference folds and satisfy additional error-rate conditions. These results do not mean that weighting or sample splitting alone guarantees validity.

Better prediction is not inherently the enemy

A population benchmark challenges a simple explanation for learner sensitivity: that flexible prediction necessarily removes useful instrument variation.

Holding the instrument and controls fixed, and assuming the treatment’s conditional mean is correctly specified, the true conditional-mean prediction of the instrument maximizes the paper’s measure of residualized strength. Thus, accurately learning that prediction has no inherent population strength penalty under these conditions.

Finite-sample results can differ. In a known-truth simulation, random-forest instrument residualization produced stronger residualized identification than ridge residualization when outcome and treatment nuisances were known correctly. With fitted linear nuisances, the ranking reversed. The comparison demonstrates why treatment-model specification must be checked before blaming flexible instrument prediction.

Evidence, diagnostics and boundaries

Across the core and additional nonlinear-nuisance simulations, Anderson–Rubin coverage ranged from 93.8% to 96.8% for nominal 95% confidence sets. But stress experiments exposed shortfalls: one weak constructed-instrument setting achieved 91.2% coverage. Accounting for clustering also mattered substantially in the reported simulations.

Five empirical applications illustrated different diagnostic patterns. The examiner application showed patterns consistent with overfitting; CollegeDistance revealed geographic leakage under shuffled folds; and a fixed-sample Card comparison showed declining conditional relevance as controls expanded. These diagnostics suggest explanations, but do not uniquely establish them or validate the instruments’ causal assumptions.

The theory assumes a homogeneous treatment coefficient and a fixed instrument dimension. Clustered guarantees rely on increasing numbers of independent clusters and sufficiently accurate nuisance estimation. They do not promise finite-sample performance with sparse constructed instruments or many instruments relative to clusters. Confidence sets can also span the entire real line: valid uncertainty need not be informative.

The practical message is to report prediction diagnostics, residualized strength and identification-robust confidence sets together—not to treat a precise-looking estimate as sufficient evidence.