用动态多尺度融合网络,提升脑电解码连续想象语音的精度。
DMF2Mel: A Dynamic Multiscale Fusion Network for EEG-Driven Mel Spectrogram Reconstruction
- 分层设计融合语义与细节特征,增强瞬态信号表达
- 对已知/未知受试者重建相关性分别达0.074和0.048,提升超35%
- 适合脑机接口、神经语音解码方向研究者参考
从脑信号中解码语音是一项挑战性研究。尽管现有技术已在词或字母级别重建听觉刺激的梅尔频谱方面取得进展,但在连续想象语音的精准重建上仍存在核心难题:传统模型难以平衡长序列解码中时序依赖建模效率与信息保留能力。为此,本文提出动态多尺度融合网络(DMF2Mel),包含四个核心模块:动态对比特征聚合模块(DC-FAM)、层次化注意力引导多尺度网络(HAMS-Net)、样条映射注意力机制(SplineMap)以及双向状态空间模块(convMamba)。DC-FAM通过局部卷积与全局注意力分离语音相关“前景特征”与噪声“背景特征”,有效抑制干扰并增强瞬态信号表示。HAMS-Net基于U-Net框架实现跨尺度语义与细节融合。SplineMap结合自适应门控科莫戈罗夫-阿诺德网络(AGKAN),融合全局上下文建模与基于样条的局部拟合。convMamba以线性复杂度捕捉长程时序依赖,并增强非线性动态建模能力。在SparrKULee数据集上的结果表明,DMF2Mel在已知受试者上达到0.074的皮尔逊相关系数(比基线提升48%),未知受试者上达0.048(提升35%)。代码见:https://github.com/fchest/DMF2Mel。
原文摘要 · Abstract (English)
Decoding speech from brain signals is a challenging research problem. Although existing technologies have made progress in reconstructing the mel spectrograms of auditory stimuli at the word or letter level, there remain core challenges in the precise reconstruction of minute-level continuous imagined speech: traditional models struggle to balance the efficiency of temporal dependency modeling and information retention in long-sequence decoding. To address this issue, this paper proposes the Dynamic Multiscale Fusion Network (DMF2Mel), which consists of four core components: the Dynamic Contrastive Feature Aggregation Module (DC-FAM), the Hierarchical Attention-Guided Multi-Scale Network (HAMS-Net), the SplineMap attention mechanism, and the bidirectional state space module (convMamba). Specifically, the DC-FAM separates speech-related "foreground features" from noisy "background features" through local convolution and global attention mechanisms, effectively suppressing interference and enhancing the representation of transient signals. HAMS-Net, based on the U-Net framework,achieves cross-scale fusion of high-level semantics and low-level details. The SplineMap attention mechanism integrates the Adaptive Gated Kolmogorov-Arnold Network (AGKAN) to combine global context modeling with spline-based local fitting. The convMamba captures long-range temporal dependencies with linear complexity and enhances nonlinear dynamic modeling capabilities. Results on the SparrKULee dataset show that DMF2Mel achieves a Pearson correlation coefficient of 0.074 in mel spectrogram reconstruction for known subjects (a 48% improvement over the baseline) and 0.048 for unknown subjects (a 35% improvement over the baseline).Code is available at: https://github.com/fchest/DMF2Mel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。