用跨模态冲突检测提升犹豫情绪视频识别准确率
Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusion
- 通过计算多模态嵌入的绝对差值，显式捕捉视觉、音频、文本间的矛盾
- 在BAH数据集上达到0.6808的宏平均F1，远超基线0.2827
- 适合关注情绪识别中不一致信号建模的研究者
我们针对CVPR 2026年第10届ABAW竞赛中的歧义/犹豫（A/H）视频识别挑战提出解决方案。方法基于跨模态分歧的多模态融合，显式度量视觉、音频和文本通道之间的冲突。视觉特征通过Py-Feat提取为动作单元（AUs），音频采用Wav2Vec 2.0，文本使用BERT。各模态经双向LSTM与注意力池化处理后投影至共享嵌入空间，融合模块计算模态间嵌入的成对绝对差值，直接捕捉表征犹豫行为的不一致性。在BAH数据集上，该方法在验证集上取得0.6808的宏平均F1，显著优于挑战基线0.2827。对1,132段视频的统计分析表明，动作单元的时间变异性是视觉上区分犹豫的核心特征。
原文摘要 · Abstract (English)
We address the Ambivalence/Hesitancy (A/H) Video Recognition Challenge at the 10th ABAW Competition (CVPR 2026). We propose a divergence-based multimodal fusion that explicitly measures cross-modal conflict between visual, audio, and textual channels. Visual features are encoded as Action Units (AUs) extracted via Py-Feat, audio via Wav2Vec 2.0, and text via BERT. Each modality is processed by a BiLSTM with attention pooling and projected into a shared embedding space. The fusion module computes pairwise absolute differences between modality embeddings, directly capturing the incongruence that characterizes A/H. On the BAH dataset, our approach achieves a Macro F1 of 0.6808 on the validation test set, outperforming the challenge baseline of 0.2827. Statistical analysis across 1{,}132 videos confirms that temporal variability of AUs is the dominant visual discriminator of A/H.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。