How an Audio Codec Could Help AI Tell Where Sounds Come From

Finding spatial clues in audio

People use differences between the sounds arriving at their two ears to judge whether a source is to the left or right. Other directional clues can be present within each ear’s signal. AI systems often handle spatial audio with a specialized encoder, separate from the machinery used to understand sound meaning.

In this preprint, Boyan Han and colleagues test another possibility: can a general-audio codec, trained without spatial supervision, provide information about both sound identity and location? They examine X-Codec, which compresses audio into layers of discrete representations. In controlled comparisons, they change factors such as the simulated room response while holding the sound source fixed. They also use linear probes—simple classifiers that test whether a property can be read from frozen features.

The authors report that the codec’s representations respond to spatial and room-related changes. Shallower layers are more influenced by sound content; deeper layers become more sensitive to differences between the ears, room acoustics and reverberation. This is a tendency, not a strict division: neither group of layers represents only one kind of information. Linear probes could recover spatial attributes, and using both channels helped most with left-right judgments. But information a probe can detect is not necessarily information a language model can use on its own.

Why retained information may be hard to use

The researchers trained a language model to answer questions about scenes containing two sound sources. In the initial setup, the audio included both sources, but the answer described only the source named in the question. The model’s left-right predictions were at chance. On other directional axes, its performance depended on whether the sources were on the same or opposite sides: same-side answers could be correct by reading the scene’s shared direction, without identifying which source was queried. On opposite-side questions, where the model had to bind a direction to the queried sound, accuracy was at chance.

The authors test two proposed explanations in a controlled 2-by-2 experiment. The first is a mismatch between audio and text: one source in the audio lacks a corresponding description. The second is that the original feature layout leaves the model to compare the two channels to recover left-right information. The researchers tested each proposed remedy separately and in combination, keeping the codec, language-model backbone, training samples and optimization procedure the same.

One remedy asked the model to describe both sources before answering. The other added explicit features representing the difference between the left and right channels. Either change lifted left-right accuracy above the original level. But on opposite-side questions, where source-specific binding matters, substantial improvement required both changes together. This supports the authors’ explanation in this experiment; it does not show that the same account applies to every spatial-audio task.

What AURA adds—and what remains uncertain

AURA combines these ideas. It groups codec layers into content-oriented and acoustics-oriented bands, adds features for the difference between channels, and trains the model to describe the scene before answering. For questions about relations between sources, its training also includes an explicit comparison of their positions. The codec and main language-model backbone stay frozen; a projector and low-rank adaptation modules are trained.

On SpatialSoundQA, a benchmark built from audio rendered with simulated room acoustics, AURA’s dual-source direction accuracy was 48.21%. The strongest compared dedicated system scored 37.90%, a difference of 10.31 percentage points. AURA used about 40% as many trainable parameters as that comparison system. These results apply to this benchmark, not to real-world hearing in general. AURA ranked second on single-source direction accuracy and relation-distance accuracy, and best on the other four reported SpatialSoundQA metrics.

The authors also report higher scores than the evaluated spatial-specialist systems on three general-audio benchmarks. General-purpose audio models scored higher on those benchmarks. These results describe the systems tested, not a universal ranking of model capabilities.

The main spatial question-answering evaluation uses simulated scenes from one benchmark. The authors also test real binaural recordings with linear probes, showing that spatial information can be read from codec features in that setting; this is not the same as demonstrating broad real-world question-answering performance. The authors note that available alternative datasets may share SpatialSoundQA’s sources or simulator rather than provide an independent test. Comparisons also have limits: systems differ in design, and one dedicated baseline was reconstructed by the authors because a complete checkpoint was unavailable.

AURA’s contribution is a demonstration that a general-audio codec can retain spatial cues, and that feature presentation and training descriptions can make those cues more useful for answering questions. Whether the approach transfers to varied recording hardware, moving sources and everyday environments remains to be tested. The source is an arXiv preprint marked under review; its findings have not yet been established through completed peer review.