Hiring models predict retention locally—but disagree across employers

A hiring platform may offer something beyond a more accurate screening algorithm: a way to compare how different employers’ models evaluate identical applicant profiles. In a preprint, Harvard Business School researchers Hossein Alidaee, Tarun Khanna and Rembrand Koning explore this possibility using data from three large employers in Mexico. Their central finding is that models with strong predictive performance inside individual firms need not provide a useful screening rule elsewhere.

The authors call the information gained from these comparisons a model comparison externality. The idea differs from pooling everyone’s data to improve one prediction system. Even when locally trained models do not transfer well, examining their differences can reveal where screening rules diverge and where applicant–employer matching might merit investigation.

A common assessment, three local models

The study covers a consumer-products company hiring across its logistics chain, a coffeehouse chain and a clothing and related goods retailer. All remain anonymous. Applicants completed the same 42-question behavioral assessment, administered through a WhatsApp chatbot. It covered seven domains, including collaboration, emotional control, diligence and workplace safety.

Two datasets played different roles. An evaluation sample contained 8,119 application records linked to hired workers whose retention beyond 90 days was observed. These data were collected before the firms adopted the platform’s screening tools. A separate production sample contained 172,242 applications submitted after adoption, with assessment responses but no observed retention outcomes used in the analysis.

Importantly, the researchers did not have access to the platform’s deployed models or their scores. They estimated their own firm-specific models using the underlying data. Each used 42 item responses and seven composite scores; age, gender and education were excluded.

After comparing five modeling approaches, the researchers selected random forests. A separate nested cross-validation exercise kept test applications outside preprocessing and tuning. Within-firm performance, measured by area under the receiver operating characteristic curve, or AUC, was 0.855, 0.891 and 0.914. These figures describe ranking performance—not the percentage of individual predictions that were correct.

Transfer performance was much weaker. Applying a model trained at one firm to observed outcomes at another produced AUCs from 0.473 to 0.514, around the 0.50 chance benchmark. Five of six confidence intervals included 0.50; the remaining estimate was below chance.

Comparing every application across firms

The researchers then scored every production application with all three models. This created a matrix showing how the same assessment profile was evaluated under each firm’s fitted retention function. Application rankings were weakly negatively correlated across models, rather than following a common ordering.

To quantify probability differences, the team compared each application’s receiving-firm prediction with its highest prediction across the three models. Giving receiving firms equal weight within assessment-index quintiles, then weighting quintiles by their production-sample shares, yielded an average gap of 21.8 percentage points. These are differences between model predictions, not observed improvements following a job move.

Some disagreement could reflect generally higher retention at one employer rather than a distinctive fit with particular profiles. The researchers therefore shifted each model’s predictions on the log-odds scale until all three had the same average predicted retention. This adjustment preserved each model’s application rankings.

The average gap fell to 15.1 percentage points. Under this normalization, differences in average prediction levels accounted for about 30% of the original gap. The remainder reflected profile-specific differences that an additive baseline shift could not remove—not a causal decomposition of why workers stay or leave.

Gaps also varied across the assessment distribution. Before adjustment, they averaged 16.4 percentage points in the bottom fifth and 26.9 points in the top fifth. After adjustment, those figures were 14.2 and 19.9 points, respectively.

Shared signals, different associations

Grouped Shapley attributions identified which assessment domains contributed to existing predictions. Collaboration had the largest average absolute contribution across the three models. But predictive importance does not establish that raising a score would improve retention.

A separate automatic debiased machine-learning analysis examined conditional retention associations for specified score increases. Emotional control supplied the clearest cross-firm differences. However, support diagnostics flagged 47 of 105 firm-specific estimates; nine of the thirteen statistically significant pairwise differences involved flagged estimates. These warnings limit confidence in much of that analysis.

The study therefore identifies differences worth testing, not a demonstrated placement strategy. It observes neither the same workers at alternative employers nor retention outcomes for the production applications. Its scope is three firms, and its score contrasts do not identify employer preferences or returns to training. Establishing whether redirecting applicants actually improves retention would require experimental assignment. For now, the contribution is showing what comparing locally useful models can reveal—and what their predictions alone cannot establish.