轻量级模型融合频谱特征,高效识别语音情感
SpectroFusion-ViT: A Lightweight Transformer for Speech Emotion Recognition Using Harmonic Mel-Chroma Fusion
- 用高效ViT架构融合音高与梅尔特征,捕捉时频模式
- 参数仅204万,计算量0.1 GFLOPs,准确率达92.56%
- 适合低资源语言部署,尤其适用于移动端和嵌入式场景
语音是表达情绪的自然方式,可靠的情感识别对人机交互、医疗、教育和客服等领域至关重要。然而,多数方法依赖重型模型或手工特征,难以在低资源语言(如孟加拉语)中兼顾精度与效率。本文提出SpectroFusion-ViT,基于轻量级EfficientViT-b0架构,仅含2.04M参数、0.1 GFLOPs计算量,通过融合音高(Chroma)与梅尔倒谱系数(MFCC)构建互补时频表示,保留精细频谱与谐波结构。经迁移学习微调后,在两个孟加拉语情感语音数据集SUBESCO与BanglaSER上分别达到92.56%和82.19%准确率,超越现有最优方法。结果表明,轻量级Transformer可在保持高效的同时实现鲁棒的情感识别。
原文摘要 · Abstract (English)
Speech is a natural means of conveying emotions, making it an effective method for understanding and representing human feelings. Reliable speech emotion recognition (SER) is central to applications in human-computer interaction, healthcare, education, and customer service. However, most SER methods depend on heavy backbone models or hand-crafted features that fail to balance accuracy and efficiency, particularly for low-resource languages like Bangla. In this work, we present SpectroFusion-ViT, a lightweight SER framework built utilizing EfficientViT-b0, a compact Vision Transformer architecture equipped with self-attention to capture long-range temporal and spectral patterns. The model contains only 2.04M parameters and requires 0.1 GFLOPs, enabling deployment in resource-constrained settings without compromising accuracy. Our pipeline first performs preprocessing and augmentation on raw audio, then extracts Chroma and Mel-frequency cepstral coefficient (MFCC) features. These representations are fused into a complementary time-frequency descriptor that preserves both fine-grained spectral detail and broader harmonic structure. Using transfer learning, EfficientViT-b0 is fine-tuned for multi-class emotion classification. We evaluate the system on two benchmark Bangla emotional speech datasets, SUBESCO and BanglaSER, which vary in speaker diversity, recording conditions, and acoustic characteristics. The proposed approach achieves 92.56% accuracy on SUBESCO and 82.19% on BanglaSER, surpassing existing state-of-the-art methods. These findings demonstrate that lightweight transformer architectures can deliver robust SER performance while remaining computationally efficient for real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。