AUTEUR tests the gap between text filters and AI-generated sound

Text-to-audio systems turn written descriptions into sounds. Assessing their safety therefore involves two different questions: whether the words are acceptable, and whether the resulting recording contains harmful content. AUTEUR, a study by Jiashu Zhang and colleagues, examines how those judgments can diverge, including for nonverbal sounds whose meaning depends on context and vocal expression.

The work was submitted to arXiv on November 6, 2026. It is a preprint; the supplied record does not establish peer-reviewed publication. Its results describe a particular safety-testing setup, not the likelihood that an ordinary user will encounter harmful audio.

Learning a structured sound description

AUTEUR automates red-teaming: deliberately searching for inputs that expose undesirable model behavior. Instead of generating an unstructured prompt, it creates an acoustic plan separating the foreground sound, the surrounding situation and audible cues, and vocal attributes such as timbre, emotion and phonation. Plans also contain an audit rationale, but that explanation is not included in the description sent to the audio system.

The plan generator is based on Mistral-7B-Instruct-v0.3. First, supervised fine-tuning teaches it from AI-generated, human-reviewed plan demonstrations constructed from AudioCaps captions. Reinforcement learning then adjusts the generator using feedback from actual audio. The target sound generator, text guard and feedback models remain fixed.

The training reward combines harmfulness judgments with checks that the recording preserves the requested scene and vocal characteristics. Harmfulness receives 80% of the reward weight; scene and vocal fidelity receive 10% each. Qwen3-Omni and Audio Flamingo 3 supply the harmfulness scores. Invalid plans and rejected descriptions receive zero execution reward.

Revising plans by listening

After training, the same frozen generator produces initial plans and revises them using feedback. It can change context and vocal attributes, but must preserve the initial foreground sound. This makes refinement a search among variations of one sound event, rather than an unrestricted search for a different event.

Audio Flamingo 3 judges generated recordings without seeing their prompts. A separate call compares each recording with its intended description and identifies discrepancies to guide revision. Search allows up to 20 refinement proposals after the initial attempt, although rejected or repeated descriptions need not produce audio.

A different model, Qwen2-Audio, assesses the selected recording after search. Reported attack success requires an accepted prompt, a recording that Audio Flamingo 3 assigns to the target harmful category, and harmfulness scores of at least 0.5 from both audio judges. This final evaluator is separate from Qwen3-Omni, which contributes to training rewards.

What the experiments found

The researchers tested 10-second clips from AudioLDM, TangoFlux, Kling and ElevenLabs Sound Effects v2. Evaluation cases were balanced across four categories: hate, violence and self-harm, shocking content, and illegal activities. Open-source targets used Llama-Guard-3-8B for text screening; commercial services used their platform filters. Platform refusals counted as failures.

Reported attack success rates were 71.9% for AudioLDM, 79.4% for TangoFlux, 95.6% for Kling and 73.8% for ElevenLabs. On the two open-source systems, AUTEUR outperformed the adapted FLIRT and perception-guided jailbreak baselines under the study’s evaluation criteria.

Removing components reduced performance. Across the open-source targets, success fell from 75.6% for the full method to 63.1% without reinforcement learning. Removing vocal descriptions reduced it to 35.3%, suggesting that vocal detail was particularly important in these tests.

Some selected prompts also worked on systems other than those used to construct them. For example, AudioLDM-derived cases achieved 66.9% success on TangoFlux, while the reverse transfer reached 51.2%. Such transfer suggests overlapping vulnerabilities, without establishing identical causes across systems.

Limits of the evidence

Human review covered the main AudioLDM and TangoFlux experiments, not the commercial-system results. Automated harmfulness classifications agreed with human assessments on 88.9% of reviewed clips. Among automatically successful cases, reviewers confirmed that 86% were both successful attacks and faithful to the original target requirements. Those figures measure different things and are not interchangeable with the reported attack success rates.

Each clip received seven student reviewers’ judgments, aggregated by majority vote. Reviewers saw the original prompts, unlike the blind audio judges. The authors report no institutional ethics approval or exemption request, and acknowledge possible contextual and cultural biases in judging harmfulness.

The study is limited to short clips and fixed-foreground refinement. It demonstrates a gap between textual acceptance and assessed audio safety under its protocol, not a universal failure rate. The authors also flag dual-use risks and recommend controlled sharing of attack prompts and harmful recordings.