融合视觉音频信息，用时序建模提升野生场景情绪预测准确率
Interactive Multimodal Fusion with Temporal Modeling
- 用ResNet和VGG提取人脸与语音特征，通过TCN建模时间动态
- 跨模态注意力让视觉与音频特征交互，融合后回归预测情绪值
- 在Aff-Wild2数据集表现优异，适合真实场景情绪分析应用
本文提出一种用于第8届野外情感行为分析竞赛（ABAW）中情绪效价-唤醒度（VA）估计的方法。该方法基于多模态框架，将视觉与音频信息融合：视觉分支采用预训练ResNet从面部图像提取空间特征；音频分支使用预训练VGG模型提取语音信号的VGGish与LogMel特征；这些特征经由时序卷积网络（TCN）进行时间建模。随后，通过交叉模态注意力机制，使视觉特征与音频特征以查询-键-值结构交互。最后，将融合后的特征拼接并输入回归层，输出情绪效价与唤醒度。该方法在Aff-Wild2数据集上取得有竞争力的表现，验证了其在真实场景下多模态融合的有效性。
原文摘要 · Abstract (English)
This paper presents our method for the estimation of valence-arousal (VA) in the 8th Affective Behavior Analysis in-the-Wild (ABAW) competition. Our approach integrates visual and audio information through a multimodal framework. The visual branch uses a pre-trained ResNet model to extract spatial features from facial images. The audio branches employ pre-trained VGG models to extract VGGish and LogMel features from speech signals. These features undergo temporal modeling using Temporal Convolutional Networks (TCNs). We then apply cross-modal attention mechanisms, where visual features interact with audio features through query-key-value attention structures. Finally, the features are concatenated and passed through a regression layer to predict valence and arousal. Our method achieves competitive performance on the Aff-Wild2 dataset, demonstrating effective multimodal fusion for VA estimation in-the-wild.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。