融合音视频信息提升野外环境下面部动作单元检测精度
Solution for 8th Competition on Affective & Behavior Analysis in-the-wild
- 用ConvNeXt和Whisper分别提取图像与音频特征
- 通过Transformer融合模块实现跨模态特征整合
- 在Aff-Wild2数据集上显著提升检测准确率
本文报告了我们在第8届野外情感与行为分析竞赛中针对面部动作单元（AU）检测挑战的解决方案。为在复杂野外环境中实现鲁棒且精准的面部动作单元分类，我们提出一种创新的音视频多模态方法。该方法采用ConvNeXt作为图像编码器，利用Whisper模型提取梅尔频谱图特征。对这些特征，我们设计基于Transformer的特征融合模块，整合音频与图像中蕴含的情感信息。该融合后的高维特征被输入至多层感知机（MLP），在Aff-Wild2数据集上进行训练，有效提升了面部动作单元检测的准确性。
原文摘要 · Abstract (English)
In this report, we present our solution for the Action Unit (AU) Detection Challenge, in 8th Competition on Affective Behavior Analysis in-the-wild. In order to achieve robust and accurate classification of facial action unit in the wild environment, we introduce an innovative method that leverages audio-visual multimodal data. Our method employs ConvNeXt as the image encoder and uses Whisper to extract Mel spectrogram features. For these features, we utilize a Transformer encoder-based feature fusion module to integrate the affective information embedded in audio and image features. This ensures the provision of rich high-dimensional feature representations for the subsequent multilayer perceptron (MLP) trained on the Aff-Wild2 dataset, enhancing the accuracy of AU detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。