When a Voice Clone Does Not Sound Like the Same Person

A voice-recognition system and a human listener can answer different questions about the same recording. The system may identify the person named in its metadata, while the listener decides whether the voice sounds like someone they have just heard. Voice cloning makes that distinction especially important: a clone is designed to reproduce a particular speaker, but listeners may still hear a different person.

Xudong Tang and Matthew Groh of Northwestern University examine this gap in VIPBench, a benchmark for voice identity perception. Their November 2026 arXiv preprint is listed in its accompanying metadata as accepted for a poster at NeurIPS 2026’s Track on Evaluations and Datasets. The work evaluates alignment with listener judgments, not whether an audio recording is real or synthetic.

Measuring perceived identity

VIPBench contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs involving 100 public figures. The speakers were sampled across five sociophonetic groups, two genders and two age brackets. Those categories are coarse sampling strata, the authors caution, rather than a complete map of variation in American English voices.

Each pair combines a reference recording with another real recording, an AI-generated clone or a synthetic blend of two speakers’ voices. The main collection includes 900 real-recording pairs, 800 clone pairs and 8,100 morph pairs. The clones and morphs were generated using Cartesia Sonic 3.

Listeners heard each pair and answered whether the recordings came from the same speaker. The researchers then calculated the proportion answering “same.” This proportion describes collective perception; it is not evidence about who actually produced a recording.

The distinction was particularly visible for clones of the reference speaker. In approximately 41% of responses to these pairs, listeners answered “different,” despite the shared target identity in the metadata. That does not mean 41% of clones were rejected by a listener majority: majority rejection occurred on 26% of those pairs, with another roughly 16% tied. Even two different real recordings of the same speaker prompted “different” responses about 31% of the time.

Recognition accuracy is not perceptual alignment

The researchers evaluated eleven publicly available speech models without fine-tuning them. Six were trained with speaker labels, four used self-supervised learning, and one—Whisper—used weak supervision for speech recognition.

Four tasks tested different aspects of alignment: predicting the proportion of “same” judgments, distinguishing listener-majority decisions and calibrating probabilities, reproducing perceived similarities between different speakers, and transferring a relationship learned from real recordings to synthetic ones. Where fitting or layer selection was needed, the protocol held out reference speakers through cross-validation.

Resemblyzer had the highest correlation with the proportion of “same” responses, at 0.656. Its squared correlation was about 0.43. Relative to the authors’ estimated noise ceiling—the variance distinguishable from chance variation in listener sampling—that amounted to approximately 61% of explainable variance, not 61% of all observed variance.

Models trained on speaker labels outperformed the self-supervised models on this continuous prediction task, while Whisper scored lowest. Yet conventional recognition performance did not determine the ordering within the supervised group. Resemblyzer’s equal error rate on the benchmark’s real-recording metadata test was 12%, compared with less than 2% for four other supervised models. Nevertheless, it aligned at least as well with listeners; differences among the top three perceptual models were not statistically significant when reference speakers were resampled.

Model rankings also depended on the question. On the task comparing perceived similarities between different speakers, most supervised models did not outperform the self-supervised group. A single leaderboard therefore would conceal meaningful differences.

Synthetic speech changes the relationship

For every evaluated model, a linear relationship fitted between model similarity and listener judgments on real recordings transferred imperfectly to clones and morphs. At high similarity scores, listeners judged synthetic pairs as the same speaker less often than the real-recording relationship predicted. The authors consequently recommend reporting real and synthetic results separately.

The benchmark has important boundaries. Its speakers are public figures speaking American English, and its real audio comes from interviews and podcasts. One commercial generator produced the main synthetic collection. An extension using CosyVoice 2 added 60 pairs from ten new speakers, but was too small to establish how broadly the synthetic-transfer findings generalize. Listener participation was also uneven, although removing the most active listeners left the main continuous-prediction ranking unchanged.

VIPBench does not demonstrate a deployed fraud defense or show that training on listener judgments improves models. It provides a way to test that research direction. Its central contribution is separating two targets that voice technology can otherwise conflate: recovering a speaker’s assigned identity and predicting the identity people actually hear.