What a Conversation’s Sound May Reveal About a Relationship

The paper introduces DyShip, a framework for classifying relationships between two speakers from their conversations. It focuses on two questions: are the speakers familiar with each other, and are they kin? These categories are related but not interchangeable. The study treats friends and coworkers as familiar non-kin, for example, while family pairs count as both familiar and kin.

The researchers used the naturalistic portion of the Seamless Interaction dataset, which provides relationship labels for recorded pairs. They used its existing training, development and test splits, with no participant shared between splits. After filtering and processing, the test set included 607 conversations from 55 dyads. Multiple conversations from the same dyad can still make observations dependent on one another.

For each task, the team compared five speech representations. Spectrograms show how sound energy changes across time and frequency; MFCCs summarize aspects of that spectral pattern. WavLM is a model that learns general-purpose speech representations. X-vectors are designed to capture speaker-discriminating information, and SPARC estimates vocal-tract movements from audio. Most representations went to a ResNet-34 classifier; x-vectors were handled with a support-vector machine. The researchers made predictions for utterances, then combined them into conversation-level predictions. They tested equal weighting and weighting utterances by duration.

The strongest individual representation depended on the label being predicted. For familiarity, spectrograms reached 86.49% accuracy with equal utterance weighting, while MFCCs had the highest area under the receiver operating characteristic curve (AUC), at 0.917. AUC measures how well a classifier ranks examples across decision thresholds; it is not the same as accuracy. For kinship, WavLM achieved 85.17% accuracy with duration-weighted averaging. X-vectors had a kinship AUC of 0.909, but their F1 score was 45.46% under the reported evaluation. That gap matters: with imbalanced classes, a strong AUC does not by itself mean a model reliably identifies kin at a chosen threshold.

Combining representations was not automatically better. Unweighted fusion produced the strongest reported results across almost all metrics, including AUCs of 0.925 for familiarity and 0.916 for kinship. The paper suggests that weaker results from weighted fusion may reflect a mismatch between development and test conditions: development used five-second utterances, while the test set used full utterances. This is a possible explanation, not something the results establish.

The authors compare their classifiers with previously reported human and audio-language-model scores on the dataset and say most classifiers exceed those benchmarks. That comparison needs care: the cited studies used different input durations, tasks and evaluation protocols. It should not be read as a direct contest under identical conditions.

There are further limits. The evaluation uses one dataset, and the development and test sets contain only 47 and 55 unique dyads, respectively. Kinship is relatively uncommon in the data, so accuracy alone can give an incomplete picture. The method classifies utterances and aggregates predictions rather than explicitly modeling turn-taking, how one speaker adapts to the other, or conversational timing. Nor do the results identify which acoustic cues drive a prediction. A system may exploit patterns associated with the labels without revealing a meaningful mechanism.

DyShip offers evidence that several kinds of acoustic information can support relationship classification in this dataset—not proof that voice reliably reveals relationships in everyday settings. The authors propose testing other datasets and modeling cross-speaker dynamics as future work. This is an arXiv preprint; the supplied source does not establish its peer-review status, and the findings and comparisons warrant independent assessment and replication.