A Low Error Rate Can Still Hide a Huge Ultrasound Measurement Mistake

A system that rarely makes large mistakes can still make an enormous one. That distinction is the central finding of a preprint by Zhengshuai Yang, Linna Jiang, Changyun Li and Guihu Zhao, who audited an AI-assisted pipeline for measuring fetal head circumference from ultrasound images. Their results concern what a particular statistical guarantee covers, rather than showing that its mathematics has failed.

The pipeline, called PUMA, separates measurement from language generation. A segmentation network identifies candidate skull contours, a geometric tool fits ellipses and converts their perimeters into millimetres, and a gate decides which candidates are eligible. A vision–language model then receives the image and proposed measurement for anatomical verification. When it accepts a measurement, it copies the tool’s value rather than estimating a new number.

A complete contour can be the wrong contour

One selection signal is contour completeness: how much of the area around a fitted ellipse’s centre is represented by contour pixels, measured across angular bins. In principle, this helps reject partial outlines. But completeness does not establish that an outline belongs to the intended anatomical structure.

The researchers found small fitted rings on bright arcs at the skull boundary. These candidates looked sufficiently complete geometrically while producing measurements far below the reference head circumference. Across the six retrained models, accepted candidates for the two highlighted images measured 51.6–54.9 mm, with completeness scores of 0.75–0.94.

The resulting errors were substantial. On one frozen test split, the largest absolute error was 271.3 mm; on the other, a different image produced a 233.3 mm error. Both occurred when the minimum permitted circumference was 50 mm.

What the certificate actually promises

The audit used the 999 labelled images in the HC18 dataset, arranged into two separately pre-registered partitions for training, model selection, calibration and frozen testing. The segmenter was retrained three times per partition. Importantly, the second partition was registered after the first failure had been identified, with its candidate measurement floors fixed before its frozen test was evaluated. It therefore served as a confirmatory replication.

The authors calibrated acceptance rules using exact Clopper–Pearson binomial bounds, with a correction for examining multiple thresholds. Their definition of a gross error was an absolute measurement error of at least 7 mm. The controlled quantity was the proportion of returned readings meeting that definition—not the maximum error or its average magnitude.

That distinction explains the apparently contradictory result. A reading wrong by hundreds of millimetres counts as one gross error, just as a much smaller error beyond the cutoff does. The paper reports calibration certificates with upper gross-error-rate bounds below 10%, including calibration partitions already containing errors above 250 mm. Such certificates do not promise that every accepted measurement is close to its reference.

Restricting size instead of demanding completeness

Raising the minimum accepted circumference from 50 to 55 mm excluded the observed small-ring failure family. The worst frozen-test errors fell to 11.4 and 11.1 mm on the two splits. Mean coverage—the fraction of images receiving a measurement—changed from 72.2% to 71.8% on the first split and from 75.0% to 74.8% on the second.

Trying to suppress the same errors by tightening the completeness threshold was considerably more costly. The authors report coverage losses of 19.0 and 12.1 percentage points. A separate, anatomically motivated 70 mm floor produced mean absolute errors of 1.91 and 1.81 mm, with coverage of 69.4% and 73.9%.

The language model was not the decisive numerical safeguard in the development benchmark either. Deterministic replay with a 50 mm floor returned 66 measurements at 1.78 mm mean absolute error, close to the delivered tool-assisted run’s 66 measurements at 1.76 mm.

These are retrospective, automated results, not clinical validation. HC18 lacks patient identifiers, so the image-level splits cannot exclude perceptual near-duplicates. Nor did the revised floor eliminate every gross error. The narrower lesson is that a guarantee about error frequency must be read alongside the size of possible failures—and that anatomical admissibility checks addressed a failure that geometric completeness alone did not.