arXiv:2511.04078cs.CV2025-11被引 4

通过语言锚定实现脑电与视觉的深层语义对齐,提升神经信号解码精度。

Unveiling Deep Semantic Uncertainty Perception for Language-Anchored Multi-modal Vision-Brain Alignment

  • 分层解耦视觉与语言语义,构建共享表征空间实现跨模态对齐。
  • 在200分类脑电检索任务中性能超越现有方法14.3%。
  • 适用于脑机接口、神经影像分析等需要高精度语义解码场景。

从脑电信号(如EEG、MEG、fMRI)中解析视觉语义仍是重大挑战,源于个体差异及视觉特征的纠缠性。现有方法多直接对齐神经活动与视觉嵌入,但仅依赖视觉表征难以捕捉潜在语义维度,限制可解释性与鲁棒性。为此,我们提出Bratrix,首个端到端实现多模态语言锚定视觉-脑对齐的框架。Bratrix将视觉刺激分解为层次化视觉与语言语义成分,将视觉与脑信号投影至共享隐空间,形成对齐的视觉-语言与脑-语言嵌入。为模拟人类感知可靠性并处理噪声神经信号,引入新颖的不确定性感知模块,实现对齐过程中的不确定性加权。通过可学习的语言锚定语义矩阵增强跨模态关联,并采用单模态预训练+多模态微调的两阶段训练策略,使Bratrix-M显著提升对齐精度。在EEG、MEG和fMRI基准测试上,实验表明其在检索、重建与图像描述任务中均优于现有最先进方法,尤其在200分类脑电检索任务中表现提升超过14.3%。代码与模型已公开。

原文摘要 · Abstract (English)

Unveiling visual semantics from neural signals such as EEG, MEG, and fMRI remains a fundamental challenge due to subject variability and the entangled nature of visual features. Existing approaches primarily align neural activity directly with visual embeddings, but visual-only representations often fail to capture latent semantic dimensions, limiting interpretability and deep robustness. To address these limitations, we propose Bratrix, the first end-to-end framework to achieve multimodal Language-Anchored Vision-Brain alignment. Bratrix decouples visual stimuli into hierarchical visual and linguistic semantic components, and projects both visual and brain representations into a shared latent space, enabling the formation of aligned visual-language and brain-language embeddings. To emulate human-like perceptual reliability and handle noisy neural signals, Bratrix incorporates a novel uncertainty perception module that applies uncertainty-aware weighting during alignment. By leveraging learnable language-anchored semantic matrices to enhance cross-modal correlations and employing a two-stage training strategy of single-modality pretraining followed by multimodal fine-tuning, Bratrix-M improves alignment precision. Extensive experiments on EEG, MEG, and fMRI benchmarks demonstrate that Bratrix improves retrieval, reconstruction, and captioning performance compared to state-of-the-art methods, specifically surpassing 14.3% in 200-way EEG retrieval task. Code and model are available.

脑机接口多模态对齐语义解码神经信号

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。