Fast, Cheap—and Fair? A Recruitment AI Audit Finds Different Answers

If an AI system changes its hiring shortlist, was it responding to a candidate’s identity—or simply producing a different answer to the same input? A new preprint examines that distinction, finding that speed, consistency, predictive quality and fairness need separate checks.

A recruitment model can be inexpensive and consistent without being fair—or particularly good at identifying suitable candidates. That is the central distinction in Fast, Cheap, and Fair? Evaluating Decision Models, a November 2026 preprint by Muhammad Uzair Ul Haq, Davide Rigoni, Alessandro Sperduti and Flavia Fabris.

The authors compare two decision models, JEV 1.13.0 and Laya, with two generative models, GPT-6 Luna and GPT-6.1 Sol. In their experiments, JEV combines lower latency and cost with shortlist quality close to the generative models. Yet measurable demographic effects remain, particularly when identity is suggested through names.

Changing identity cues, not qualifications

Decision models return probabilities over predefined answers rather than generating an answer token by token. Here, all four systems evaluated the same seven recruitment criteria, including required skills, experience, seniority and education. Their assessments were combined into a score from 0 to 100. The rubric did not instruct models to ignore demographic information.

For the bias audit, the researchers used 572 English-language CVs across 11 occupations, each paired with an occupation-specific job description. They inserted either an explicit demographic statement or a name intended to signal gender, ethnicity or geographic origin, leaving the rest of each CV unchanged.

Each altered CV was rescored while competitors’ scores stayed fixed. The researchers examined score differences, ranking movements and exclusions from the original top-five shortlist. Separately, they scored unchanged CVs seven times to establish how much decisions varied without any demographic modification.

That second measurement matters: a changed shortlist does not, by itself, show that a demographic cue caused the change.

Instability can obscure demographic effects

On unchanged inputs, JEV’s average per-CV score standard deviation was 0.50 points, compared with 4.11 for GPT-6 Luna and 1.43 for GPT-6.1 Sol. Laya returned identical results across repeated runs.

The shortlist measure told a similar story. Averaged across runs, the proportion of candidates from a reference top-five shortlist missing from an individual run was 2.9% for JEV, 29.6% for GPT-6 Luna and 18.3% for GPT-6.1 Sol. The reference shortlist used candidates’ mean scores across all seven runs.

Neither generative model showed statistically significant demographic score or rank effects in the audit. The authors caution that this does not establish an absence of bias: larger background variation can make systematic differences harder to detect.

JEV showed small but statistically significant effects. Its largest average demographic score difference was 0.37 points on the 100-point scale, while average ranking movements stayed below 0.4 positions in pools of 41–65 CVs. Across the tested conditions, exclusions from original top-five shortlists reached 8.2%, compared with its 2.9% run-to-run disagreement measure.

Names sometimes mattered more than explicit declarations. Asian-associated names, for example, reduced JEV scores by 0.11 points relative to White-associated names; explicitly stating Asian ethnicity had little effect. The authors also stress that ranking and exclusion measures compare altered CVs with originally nameless CVs, so they capture both adding a name and any group-specific effect.

Consistency is not predictive quality

The researchers assessed screening quality on a separate labeled dataset, JD-Fit, using 1,759 test pairs from 71 job descriptions. Model-specific decision thresholds were selected using separate training data.

JEV’s Precision@5—the share of shortlisted CVs labeled suitable—was 72.4%, compared with 72.5% for GPT-6 Luna and 71.5% for GPT-6.1 Sol. The dataset’s random-ranking baseline was 57%. Neither generative model significantly outperformed JEV on AUC, a measure of how well scores distinguish suitable from unsuitable pairs across thresholds.

Laya illustrates why repeatability alone is insufficient. Despite identical repeated outputs, its AUC was 0.520, not significantly above chance, and its Precision@5 was 59.9%. Small CV modifications also changed its rankings and shortlists.

JEV’s efficiency advantage was substantial in this setup. On JD-Fit, median latency was 0.30 seconds per call, versus 9.91 seconds for GPT-6 Luna and 7.17 seconds for GPT-6.1 Sol. Estimated costs per 1,000 calls were $0.14, $0.73 and $9.32, respectively. Laya ran locally, so the study reports no API charge for it.

An audit, not a fairness certificate

The authors explicitly say these results do not certify any model as fair for hiring. The bias dataset covers only 11 occupations, uses binary gender categories and excludes disability, religion, socioeconomic status and intersectional groups. Name choices follow US and UK conventions and can convey more than the intended identity cue.

The experiments measure responses to controlled CV edits, not downstream hiring outcomes. The authors recommend domain-specific audits with real applicant data, renewed evaluation after model updates and continued human oversight.