用听觉脑电信号实现论文检索，让失明者也能用脑波搜文献。
Cross-Sensory Brain Passage Retrieval: Scaling Beyond Visual to Audio
- 用脑电波直接映射文本，跳过文字转换步骤。
- 听觉脑信号检索效果优于视觉，联合训练提升31%准确率。
- 适合残障用户和语音交互场景，突破数据少的限制。
由于认知复杂性和身体障碍，从内部信息需求生成查询在所有信息检索范式中仍具根本挑战。脑段落检索（BPR）通过直接将脑电图（EEG）信号映射到段落表示，绕过中间文本转换。然而现有BPR研究仅使用视觉刺激，未解决关键问题：听觉EEG能否支持语音接口与视障用户的高效检索？跨感官训练是否能缓解严重数据稀缺并提升性能？本文首次系统研究听觉EEG用于BPR，并评估跨感官训练收益。采用双编码器架构与四种池化策略（CLS、均值、最大值、多向量），在Alice（听觉）和Nieuwland（视觉）数据集上对比纯听觉、纯视觉及联合训练。结果表明：听觉EEG持续优于视觉EEG；跨感官训练结合CLS池化，在MRR（0.474）、Hit@1（0.314）和Hit@10（0.858）上分别较单一训练提升31%、43%、28%；联合模型超越BM25文本基线（MRR: 0.474 vs 0.428），证明神经查询可媲美传统检索，同时支持无障碍访问。研究验证了听觉神经接口在信息检索中的可行性，并表明跨感官训练有效应对数据稀缺，优于单模态方法。
原文摘要 · Abstract (English)
Query formulation from internal information needs remains fundamentally challenging across all Information Retrieval paradigms due to cognitive complexity and physical impairments. Brain Passage Retrieval (BPR) addresses this by directly mapping EEG signals to passage representations without intermediate text translation. However, existing BPR research exclusively uses visual stimuli, leaving critical questions unanswered: Can auditory EEG enable effective retrieval for voice-based interfaces and visually impaired users? Can training on combined EEG datasets from different sensory modalities improve performance despite severe data scarcity? We present the first systematic investigation of auditory EEG for BPR and evaluate cross-sensory training benefits. Using dual encoder architectures with four pooling strategies (CLS, mean, max, multi-vector), we conduct controlled experiments comparing auditory-only, visual-only, and combined training on the Alice (auditory) and Nieuwland (visual) datasets. Results demonstrate that auditory EEG consistently outperforms visual EEG, and cross-sensory training with CLS pooling achieves substantial improvements over individual training: 31% in MRR (0.474), 43% in Hit@1 (0.314), and 28% in Hit@10 (0.858). Critically, combined auditory EEG models surpass BM25 text baselines (MRR: 0.474 vs 0.428), establishing neural queries as competitive with traditional retrieval whilst enabling accessible interfaces. These findings validate auditory neural interfaces for IR tasks and demonstrate that cross-sensory training addresses data scarcity whilst outperforming single-modality approaches Code: https://github.com/NiallMcguire/Audio_BPR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。