用音频频率对齐脑电与语音信号,提升听觉注意力检测精度
Frequency-Based Alignment of EEG and Audio Signals Using Contrastive Learning and SincNet for Auditory Attention Detection
- 结合SincNet与对比学习,对齐脑电信号与语音特征
- 在KUL和DTU数据集上分别达78.3%和92.2%准确率,1秒内完成判断
- 仅用6个颞叶电极即可媲美64电极,适合低密度设备部署
人类在复杂声学环境(如鸡尾酒会)中具有出色聚焦听觉注意力的能力。听觉注意力检测(AAD)旨在通过分析脑电图(EEG)等脑信号识别出正在关注的说话人。现有方法多依赖深度学习的非线性建模能力,却较少考虑大脑听觉处理的神经机制。本文提出SincAlignNet,一种基于改进SincNet与对比学习的新网络,用于对齐音频与EEG特征以实现听觉注意力检测。SincNet组件模拟大脑在听觉注意过程中的音频处理机制,而对比学习引导模型学习脑电信号与关注语音之间的关联。推理时,计算脑电与音频特征间的余弦相似度,并探索直接从脑电推断关注说话人。跨试验评估显示,SincAlignNet在两个公开数据集KUL和DTU上分别达到78.3%和92.2%的平均准确率,决策窗口为1秒。模型具有强可解释性,揭示左右颞叶在男女说话场景中均更活跃。此外,仅使用靠近颞叶的6个电极即可获得与64电极相当甚至更优的性能,表明高效低密度EEG在线解码是可行的,为神经引导助听器在真实场景中的应用迈出关键一步。代码已开源:https://github.com/LiaoEuan/SincAlignNet。
原文摘要 · Abstract (English)
Humans exhibit a remarkable ability to focus auditory attention in complex acoustic environments, such as cocktail parties. Auditory attention detection (AAD) aims to identify the attended speaker by analyzing brain signals, such as electroencephalography (EEG) data. Existing AAD algorithms often leverage deep learning's powerful nonlinear modeling capabilities, few consider the neural mechanisms underlying auditory processing in the brain. In this paper, we propose SincAlignNet, a novel network based on an improved SincNet and contrastive learning, designed to align audio and EEG features for auditory attention detection. The SincNet component simulates the brain's processing of audio during auditory attention, while contrastive learning guides the model to learn the relationship between EEG signals and attended speech. During inference, we calculate the cosine similarity between EEG and audio features and also explore direct inference of the attended speaker using EEG data. Cross-trial evaluations results demonstrate that SincAlignNet outperforms state-of-the-art AAD methods on two publicly available datasets, KUL and DTU, achieving average accuracies of 78.3% and 92.2%, respectively, with a 1-second decision window. The model exhibits strong interpretability, revealing that the left and right temporal lobes are more active during both male and female speaker scenarios. Furthermore, we found that using data from only six electrodes near the temporal lobes maintains similar or even better performance compared to using 64 electrodes. These findings indicate that efficient low-density EEG online decoding is achievable, marking an important step toward the practical implementation of neuro-guided hearing aids in real-world applications. Code is available at: https://github.com/LiaoEuan/SincAlignNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。