融合多模态语音特征与上下文注意力,提升帕金森病检测精度
Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention

- 构建三路分支网络,分别处理频谱图、梅尔倒谱系数和HuBERT嵌入
- 在西班牙PC-GITA数据集上达91.51%准确率,AUC达95.97%
- 通过上下文引导的跨模态注意力机制实现多特征动态融合,适合临床辅助诊断
帕金森病(PD)是一种进行性神经退行性疾病,常伴随运动减少性构音障碍。由于言语产生依赖复杂的神经肌肉协调,语音分析已成为早期无创、低成本生物标志物的有力候选。现有深度学习方法多依赖单一语音表征,可能遗漏不同特征空间中的互补病理信息。本文提出一种多分支深度学习框架,从语音中自动检测帕金森病。每段语音被分割为5秒片段,采用三种互补模态表示:对数梅尔频谱图、梅尔频率倒谱系数(MFCCs)和从原始波形提取的HuBERT嵌入。频谱图通过预训练的ResNet-18编码器处理,MFCC序列由双向LSTM建模,原始语音则使用预训练HuBERT模型编码。为有效融合异构表征,引入上下文引导的跨模态注意力机制,根据频谱图和MFCC分支提取的全局声学上下文,动态加权时间维度上的HuBERT嵌入。在公开的西班牙PC-GITA语料库上,采用严格的说话人无关五折交叉验证,所提方法取得91.51%准确率、91.24% F1分数和95.97% AUC。消融实验进一步证实了该注意力机制和多模态融合的有效性。结果表明,异构语音建模在鲁棒且临床可靠的帕金森病检测中具有巨大潜力。
原文摘要 · Abstract (English)
Parkinson's disease (PD) is a progressive neurodegenerative disorder that frequently causes speech impairments associated with hypokinetic dysarthria. As speech production relies on the precise coordination of complex neuromuscular mechanisms, speech analysis has emerged as a promising non-invasive and cost-effective biomarker for early PD detection. Recent deep learning approaches have shown encouraging results; however, most existing methods rely on a single speech representation, potentially overlooking complementary pathological information encoded across different feature spaces. In this work, we propose a multi-branch deep learning framework for automatic PD detection from speech. Each recording is segmented into 5-second chunks and represented using three complementary modalities: Log-Mel spectrograms, MFCCs, and HuBERT embeddings extracted from raw waveforms. The spectrograms are processed using a pre-trained ResNet-18 encoder, MFCC sequences are modeled through a BiLSTM network, and raw speech is encoded using a pre-trained HuBERT model. To effectively integrate these heterogeneous representations, we introduce a context-guided cross-modal attention mechanism that dynamically weights temporal HuBERT embeddings according to the global acoustic context derived from the spectrogram and MFCC branches. Experiments conducted on the publicly available Spanish PC-GITA corpus under strict speaker-independent 5-fold cross-validation demonstrate the effectiveness of the proposed approach. The proposed architecture achieves an accuracy of 91.51%, an F1-score of 91.24%, and an AUC of 95.97%. Furthermore, ablation studies confirm the contribution of both the proposed context-guided cross-modal attention mechanism and the integration of complementary speech representations. These findings highlight the potential of heterogeneous speech modeling for robust and clinically reliable PD detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。