Using two-channel hearing to identify which object is making a sound

From recognizing a category to identifying an object

When several phones appear in an image, recognizing a ringtone may identify the relevant object category without revealing which phone produced it. Audio-visual segmentation systems can consequently highlight multiple same-class objects instead of isolating the sounding one.

In the preprint Which One Is Sounding? Binaural Sounding Instance Disambiguation for Audio-Visual Segmentation, Haoze Ma and colleagues propose BiSID-AVS to address this ambiguity. Their setting combines a single visual observation with synchronized binaural audio: separate left and right channels containing spatial information about the source.

The authors distinguish this problem from video-based instance segmentation, where motion and associations across frames can help preserve object identity. Those cues may be unavailable with sparse observations or objects that barely move. BiSID-AVS instead uses differences between the two audio channels to help determine which visible candidate is sounding.

Spatial evidence shapes the mask

The method separates sound-category information from spatial information before bringing them together during segmentation.

First, it averages the audio channels and uses a pretrained VGGish audio encoder to produce semantic features. These features prompt the Segment Anything Model, or SAM, to represent image regions consistent with the sound category. This stage establishes plausible candidates rather than deciding which candidate is responsible.

A second pathway keeps the audio channels separate. It converts each waveform into a cochleagram, a representation of sound across frequency and time, and processes the pair into a spatial query. Comparing that query with unprompted visual features and image-position information produces a dense spatial response map.

Finally, the response map conditions both the candidate features and SAM’s mask decoder. The authors’ central design choice is to introduce spatial evidence while the mask is being formed, rather than generate independent visual masks and choose one afterward. Training also explicitly penalizes activation on silent objects belonging to the same category.

Large gains in the ambiguous cases

The researchers construct two benchmarks. Binaural AVS-Synthetic contains 31,045 samples across 46 categories, with 14.5% containing multiple same-class candidates. Its audio is synthesized using head-related acoustic filters, with source direction derived from image position and distance informed by estimated relative depth.

Habitat-AVS contains 11,781 samples across 14 categories, four simulated indoor scenes and ten predefined robotic viewpoints. Here, 45.3% of samples contain multiple same-class candidates, and audio is rendered from the sounding object’s three-dimensional position.

On the multi-candidate subsets, the authors report mean intersection-over-union scores of 70.01% for Binaural AVS-Synthetic and 89.52% for Habitat-AVS. This metric measures overlap between predicted and reference object masks. The strongest comparison scores were 34.82% and 48.31%, making the reported improvements 35.19 and 41.21 percentage points, respectively.

The gains are not universal. On single-candidate samples, BiSID-AVS does not achieve the highest scores; the authors note that spatial conditioning offers limited benefit there and can slightly reduce mask completeness.

The comparisons also reflect the paper’s particular task. Monaural baselines receive channel-averaged audio, while video methods receive repeated copies of a static image. The authors explicitly caution that the adapted AVISM results are not an upper bound on its performance with dynamic video.

What the controls and demonstration show

Audio perturbations provide evidence that the system uses spatial correspondence. On Habitat-AVS multi-candidate samples, swapping the left and right channels lowers mean mask overlap from 89.52% to 17.37%. Substituting audio from another source location in the same scene and category lowers it to 11.41%, with the image unchanged.

A separate evaluation holding out entire scenes reports 85.43% multi-candidate overlap for BiSID-AVS, compared with 45.37% for SAMA-AVS. The authors also demonstrate a mobile robot identifying a sounding target among three same-class devices and converting depth within its predicted mask into a navigation goal. This demonstrates feasibility, rather than a quantified navigation success rate.

Important limits of the evidence

These are preprint findings from constructed benchmarks. Habitat-AVS disables indirect reflections and material-dependent acoustics, emphasizing source–receiver geometry. Its multi-candidate samples contain two same-class objects, and generation checks require non-overlapping object bounding boxes. Those conditions constrain what the benchmark establishes about more complicated scenes.

Most importantly, the formulation assumes one active source per audio segment. Distinguishing multiple objects producing overlapping sounds remains future work. The reported results support binaural spatial conditioning for the studied single-source ambiguity, not unrestricted sound understanding in everyday environments.