AudioSphere learns sound identity and direction through masked audio training

Many general-purpose audio models learn from single-channel recordings, which lack the spatial information available from microphone arrays. AudioSphere instead learns from multichannel sound fields, aiming to preserve both what is happening and where sounds originate. In a November 6, 2026 preprint, Goksenin Yuksel, Marcel van Gerven and Kiki van der Heijden report that its learned features transfer to conventional audio tasks and sound event localization and detection, or SELD.

The distinction between learning features and learning a task is important. AudioSphere’s pre-training does not require event or direction labels. Downstream evaluation still trains prediction heads using labeled examples. A “frozen” backbone means the main feature extractor remains unchanged, not that the entire system operates without supervised training.

Reconstructing sound and spatial cues

AudioSphere uses first-order Ambisonics, a four-channel representation containing an omnidirectional component and three directional components. This provides a common input format rather than tying the model to one microphone layout.

Its inputs combine four log-mel spectrograms, which describe how frequency content changes over time, with three active intensity-vector components. These vectors summarize directional information derived from relationships between the audio channels.

During pre-training, the system hides 80% of randomly selected time-frequency patch locations across all seven input channels simultaneously. A Transformer encoder processes the visible patches, and a decoder reconstructs the missing acoustic and directional features. Hiding matching locations across channels prevents the model from simply copying information from an unmasked counterpart. Ablation experiments support this joint masking approach over masking channels separately, although the chosen masking ratio is not best on every reported metric.

The encoder subsequently produces features at 80-millisecond intervals. Aggregating features also provides a representation for an entire clip, allowing the same backbone to support both time-resolved predictions and clip classification.

Spatial training built largely through simulation

Because large native Ambisonics collections are scarce, the researchers spatialized approximately 1.74 million AudioSet clips, representing about 4,800 hours of source audio. They simulated 70,000 room impulse-response sets across 70 Matterport3D houses, then combined sound clips with localized or diffuse background noise.

The simulations used stationary sources. Nevertheless, the model was later tested on moving-source recordings because its features retain spatial information over short intervals. Training sampled 16 partially overlapping two-second segments per clip to reuse the costly simulated mixtures; these crops should not be confused with additional independent recordings.

The strongest aggregate transfer to single-channel tasks came from mixing 75% original, non-spatialized AudioSet examples with 25% simulated spatial examples. Original clips were duplicated across channels to fit the input format.

What the evaluations show

On the paper’s selected 11-task HEAR evaluation, covering environmental sounds, speech, music and event detection, AudioSphere achieved an aggregate score of 83.2, with a standard deviation of 0.1 across three downstream training seeds. Its clean-audio counterpart scored 83.0 ± 0.2, suggesting broadly comparable transfer rather than a clear advantage from spatial training alone.

AudioSphere’s aggregate exceeded the compared models, including Dasheng at 82.8. However, Dasheng retained stronger speech results. The aggregate is not a universal accuracy measure, and some baseline rows lack results for individual tasks. The reported seed variability also measures downstream probe training, not independent repetitions of backbone pre-training.

The researchers additionally introduce RealSELD, an evaluation kit organizing five existing datasets into seven tasks. Most datasets combine isolated sounds with measured room responses; STARSS23 includes direct recordings of natural indoor scenes. These are different levels of realism, not interchangeable forms of field testing.

On stationary TAU-2019 scenes, frozen AudioSphere features with a shallow prediction head achieved a detection F-score of 84.6 and localization error of 14.1 degrees, versus 79.9 and 28.5 degrees for the supervised SELDNet baseline. SELDNet nevertheless retained higher localization recall, illustrating why angular error alone is insufficient.

Moving, overlapping sources exposed a limitation. On TAU-2021, the frozen model’s shallow head scored 21.0 on location-dependent F-score, below the baseline’s 30.7. Adding a trainable temporal head raised this to 37.2; fine-tuning both backbone and temporal head reached 57.7. This metric requires the correct sound class and a direction within 20 degrees of the reference.

Hardware flexibility has limits

Ambisonics Signal Matching converts other microphone layouts into the model’s input format using fixed filters based on array geometry. On planar-array RSL2019-dev recordings, the complete AudioSphere pipeline achieved a median direction error of 1.9 degrees, compared with 5.6 for MUSIC and 13.1 for the evaluated Spatial-AST pipeline. These comparisons do not isolate encoder quality; front–back ambiguity explains part of the Spatial-AST gap.

Conversion cannot recover spatial detail the microphones never captured, and tetrahedral-array conversion worsened localization relative to reference Ambisonics. Complex motion likewise benefits from temporal modeling. The study evaluates detection and direction—not distance, source separation or dereverberation. Its preprint results support reusable spatial-acoustic features, not complete three-dimensional sound understanding.