CAST-Bench tests whether AI-generated sounds belong to the right scene and speaker

A barking dog in a generated soundtrack does not, by itself, mean a video generator has followed its instructions. The bark could occur in the wrong shot or accompany the wrong animal. CAST-Bench approaches such errors through an “audiovisual contract”: requested sounds should be present, associated with their intended sources, correctly timed and consistent with the story.

The benchmark is described in the preprint CAST-Bench: Tracing Every Sound to Its Source in Multi-Shot Audio-Video Generation, submitted to arXiv on November 2, 2026. Yucong Liu, Qi Tian, Shuyuan Tu, Kaihang Pan, Weijie Kong and Chao Pan present it as a diagnostic evaluation framework, rather than a new generation model.

Building explicit requirements

The suite contains 900 prompts, divided equally among nine categories: speech, non-verbal vocalizations, animals, objects, actions, ambient sounds, instruments, background music and sound effects. There are 439 single-shot prompts and 461 multi-shot prompts; the latter request between two and nine shots.

Each prompt is paired with a structured specification recording subjects, sound events and shot requirements. Generators receive the natural-language prompt; evaluators also receive its annotation. This separates producing a sound somewhere in a clip from producing it in the specified context.

Domain experts establish the taxonomy and controlled vocabularies. GPT-5.5 fills structured specifications and converts them into prompts, followed by structural and semantic checks, duplicate checks and coverage inspection. The resulting prompts have English and Chinese versions. Speech requirements span nine languages, although English and Chinese account for 79 of the speech category’s 100 prompts. Equal representation of sound categories therefore does not mean equal representation of every language or other condition.

What the scores can establish

CAST-Bench reports 18 metrics across visual performance, content consistency, and audio and semantic alignment. Specialist systems assess properties including visual quality, recognized text, face stability, speech transcription and audiovisual timing. Qwen3-Omni and Qwen3-VL judge requirements such as audible event presence, visual subject correctness and dialogue assignment using structured questions or scoring rubrics.

The distinctions matter. Audio quality is evaluated separately from whether requested sounds are audible. Dialogue transcription is evaluated separately from whether each line belongs to the correct speaker. Speaker assessment first constructs a speaker-aware transcript, then compares identities and utterances with the prompt.

Despite its name, the Sound-Source Match metric measures temporal offset between audio and corresponding visual activity. The authors explicitly note that it does not verify the source assignment of every sound. The benchmark’s broader contract is therefore investigated through complementary measurements, not certified by one comprehensive sound-tracing test.

Different shots, different results

The researchers evaluated 12 generators on single-shot prompts and nine on multi-shot prompts. Ovi, LTX-2.3 and daVinci-MagiHuman were excluded from the latter because they do not support multi-shot generation. Each condition used three independent seeds or repeated API calls.

Generation settings were not entirely identical. Most outputs were nominally 10 seconds at 720p, while Veo3.1 used eight seconds and daVinci-MagiHuman its native 540p. Kling models received Chinese prompts because of input-length limits; other models received English prompts.

Seedance-2.5, Seedance-2.0 and MiniMax-H3 led both leaderboards. Their single-shot overall scores were closely grouped at 78.11, 78.03 and 77.97, respectively. Across the nine shared models, the mean overall score was 74.30 for single-shot prompts and 63.59 for multi-shot prompts.

That 10.71-point difference needs qualification. The two subsets use different metric compositions and weights: shot-count matching contributes only to multi-shot scoring. The gap describes performance under those respective rules, not an isolated causal effect of adding cuts. Nor did every measurement worsen: sound-source synchronization and style matching improved on average, even as every shared model’s aggregate score fell.

Dialogue and continuity remain difficult

Qualitative inspection found concurrent dialogue being serialized, one speaker’s line disappearing, or multiple faces mouthing the same words. Other failures included clipped dialogue, unrequested extra speech and ambient sounds becoming faint or missing during speech.

Within the speech subset, all nine shared models scored lower on multi-speaker than one-speaker prompts in both shot settings. They also scored lower on multi-shot than single-shot prompts within each speaker-count group. These comparisons identify recurring difficulties without establishing that speaker count alone caused them.

For human comparison, three annotators independently assessed each sampled video, with 50 videos sampled per model. Automatic and averaged human overall scores had video-level Spearman rank correlations of 0.833 for single-shot and 0.784 for multi-shot outputs. These are point estimates of agreement in ranking videos, not percentages of correct judgments.

The study’s central distinction is between convincing presentation and fulfilled instructions. High visual scores do not establish that the right words, sounds and speaker relationships survive a sequence of shots. CAST-Bench makes those requirements more explicit while retaining important limits on what its individual scores demonstrate.