Why a better-fitting choice model may not make better forecasts

Two kinds of prediction

Choice models estimate the probabilities of decisions, such as travelling by bus rather than car. Fixed-coefficient models use a common set of preference coefficients, potentially adjusted through observed characteristics. Mixture models instead represent unobserved differences between people, either through a continuous distribution of preferences or through distinct groups called latent classes.

When researchers observe repeated choices from each person—panel data—mixtures often fit much better. But Stephane Hess and Sander van Cranenburgh argue that much of this improvement answers a different question from the one forecasters usually ask.

An unconditional forecast uses the estimated population distribution of preferences, plus any relevant observed characteristics, without using that person’s other choices. A conditional forecast updates the preference distribution using the person’s observed decisions. Someone’s earlier trips, for example, can help predict their next travel mode.

What better fit is rewarding

The paper’s central mathematical result is that a mixture’s panel log-likelihood, its estimation objective, equals the accumulated log score of forecasts that condition each choice on the person’s preceding choices. An unconditional forecast is scored by a different quantity: the cross-sectional log-likelihood, which evaluates choices separately.

The authors call the difference between these scores, evaluated at the same model parameters, choice consistency. It measures the benefit of representing dependence between a person’s choices. It does not establish that earlier decisions cause later ones: the panel likelihood itself is unchanged by reordering the choices.

Their illustrative example makes the distinction concrete. Suppose two equally common groups choose an option with probabilities of 0.9 and 0.1. For someone whose group is unknown, the predicted probability is 0.5—the same prediction a model without heterogeneity could make. Yet the mixture predicts that someone will repeat the same choice twice with probability 0.82, rather than 0.5 under independent predictions.

The authors split a panel mixture’s fit improvement over a comparable fixed-coefficient model into choice consistency and a remainder called realised heterogeneity. The remainder is the improvement in scoring individual choices without using personal choice histories. It can be negative.

Evidence from travel and vaccine choices

The researchers examined 11,022 recorded trips from 373 people and 12,882 hypothetical vaccine choices from 2,147 UK respondents. Estimation used a median of 23 trips per traveller and five tasks per vaccine respondent.

They tested seven mixture specifications per dataset, each estimated using both panel and cross-sectional likelihoods. Five-fold validation held out entire individuals; additional holdouts reserved later choices from people included in estimation.

Across all fourteen panel mixtures, choice consistency accounted for more than 90% of the fit gain. The remaining share ranged from −18% to 8%, using averages across folds. It was negative for all seven travel specifications and small but positive for all seven vaccine specifications.

For travel, the mixed logit improved panel fit over multinomial logit by about 986 log-likelihood points. Nevertheless, for held-out individuals it assigned the chosen alternative a mean probability of 0.739, compared with 0.778 for multinomial logit, and had a worse log score. These probabilities are not percentages of choices classified correctly.

Shortening the travel panels changed the picture. The single-choice score advantage turned negative between ten and fifteen estimation trips per person for mixed logit, and later for latent class. This panel-length finding was tested within that dataset and those two model families, not established universally.

Why the estimates can diverge

The authors explain that, under misspecification, matching dependence between repeated choices can pull estimated preference differences away from what best describes individual choices. Habit, changing preferences or an unsuitable distribution could create such a mismatch. The study detects misspecification but does not establish its source.

This is not an inevitable defect of mixtures. With correct specification, sufficient identification from single-choice probabilities and large samples, both estimation approaches target the same distribution; panel estimation is more efficient. Nor must distortion always widen estimated preference distributions: different coefficients can move in different directions.

When personal history helps

Conditioning reversed the practical comparison. For travellers included in estimation, using their earlier trips raised mixed logit’s mean probability for the chosen alternative on later trips from 0.735 to 0.857. For the vaccine study’s final task, conditioning raised the corresponding probability from about 0.33 to 0.46.

An important exception concerned aggregate prediction: panel mixtures recovered market shares much better on the vaccine study’s unusual fixed final task. The authors caution against generalising from that single scenario, which was always presented last.

Their recommendation is straightforward: report a panel mixture’s unconditional single-choice score alongside its fit. This needs one additional likelihood evaluation, not re-estimation. It reveals which part of the fit improvement is relevant to forecasting without personal histories, although validation on unseen data remains necessary.