BrainBind compares EEG and fMRI video retrieval—and exposes limits on unseen episodes

A shared reference for brain signals

BrainBind, a November 3, 2026 arXiv preprint by Shixiao Wang, Rining Wu and Jian K. Liu, compares synchronized EEG and fMRI recordings collected during long-form naturalistic viewing. The authors investigate how these signals can identify matching video segments through pretrained video representations. The reported evidence concerns retrieval within CineBrain, not unrestricted reconstruction of what someone sees.

EEG and fMRI capture complementary aspects of brain activity: EEG offers fast temporal measurements but weak spatial localization, while fMRI provides spatial information through slower hemodynamic responses. BrainBind maps these recordings into a frozen video embedding space—a numerical representation used to compare clips. The authors explicitly treat that space as a visual-semantic reference, not a literal neural code.

Comparing two anchors and four systems

The main experiment crosses two frozen video models, Omni and ImageBind, with four neural configurations: EEG-only, fMRI-only, cross-attention fusion called Both, and Joint-Dual. The first three use Transformer encoders. Joint-Dual uses two structured state-space encoder branches, mean fusion, and contrastive objectives for both individual modalities and their combined representation.

The main evaluation uses a fixed 60/20/20 random sample split and a 6,480-pair evaluation pool. Retrieval runs in both directions: brain signals retrieve video candidates, and video representations retrieve brain recordings. Candidates sharing the same stimulus label count as positives, so success does not require selecting one unique recording.

The authors report five-seed recall results for all eight configurations. Most additional ranking metrics also use five seeds; EEG–ImageBind uses four because one run lacks those outputs.

The first result and the broader ranking differ

With Omni, Joint-Dual achieves the highest mean top-choice recall in the main matrix: 13.52%, compared with 12.71% for fMRI-only. These figures average brain-to-video and video-to-brain retrieval. They measure how often a label-positive candidate appears first, not video reconstruction fidelity.

The fMRI-only system performs better on broader ranking measures, reaching mean reciprocal rank of 23.82%, mean average precision of 21.37%, and nDCG@10 of 25.31%. Joint-Dual reaches 18.69%, 15.09%, and 16.52%, respectively. The authors describe this as a precision–breadth trade-off: more first-place successes do not necessarily mean better organization of positives throughout the ranked list.

That comparison does not isolate the benefit of adding EEG. Joint-Dual–Omni has about 294 million parameters versus approximately 50 million for the fMRI-only Transformer, and its architecture and objective also differ. The authors identify capacity-matched controls as necessary future work. Their seed-level statistical tests use unadjusted p-values.

Unseen episodes reveal context dependence

The random split is a major interpretive constraint. Assuming approximately four observations per stimulus, the authors estimate that 93.6% of test samples share their exact segment identity with training data. Random-split performance therefore measures retrieval largely within recurring stimulus and narrative contexts.

A stricter experiment keeps all clips from each season–episode group in one partition. Across three runs coupling model initialization and episode-assignment seeds, mean top-choice recall falls to 0.075% for EEG, 0.129% for fMRI, 0.126% for Both, and 0.216% for Joint-Dual. The reported label-aware chance floor is 0.069%.

Joint-Dual remains highest, but absolute performance is low. These results support the authors’ conclusion that repeated stimulus identity and episode continuity contribute materially to random-split retrieval. They do not establish robust generalization to unfamiliar narratives.

Proxy choice matters, but causation remains unresolved

Omni outperforms ImageBind across corresponding configurations. The authors also find that Omni’s embeddings distribute variance across more dimensions, consistent with its stronger retrieval results. This is an observational association, not proof that the broader geometry causes better alignment.

Participant generalization is similarly unsettled. Selective checks cover only two held-out participants, and checkpoint selection still uses some data from each. Joint-Dual’s mean top-choice recall differs sharply between them: 24.150% and 0.069%. These checks cannot provide a population-level estimate.

Finally, the generated-video examples use retrieved text descriptions to prompt a fixed generator. They are qualitative, retrieval-conditioned visualizations—not direct brain reconstructions. The preprint’s central contribution is therefore a comparison framework whose results expose both retrieval differences and substantial generalization limits.