融合文本、音频、视觉多模态信息，提升视频中犹豫与矛盾情绪识别准确率。
Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion
- 用跨模态注意力和门控融合机制联合处理三类模态数据
- 在验证集上达到0.7394的宏平均F1，比单模态最佳结果提升11%
- 适合关注多模态情感分析、人机交互的研究者使用
我们提出一种用于视频中犹豫与矛盾（A/H）识别的多模态框架，专为ECCV 2026 ABAW11挑战设计。采用BAH数据集，通过F2LLM-v2-0.6B（1024维文本）、WavLM-Large（1024维音频）和VideoMAE V2（768维面部视频）三个预训练编码器提取特征。先建立单模态基线，使用MLP、随机森林、GBDT等经典分类器，经Optuna优化后，仅用文本特征即达0.6659的测试集宏平均F1，显著优于零样本Video-LLaVA基线（0.2827）。在此基础上，提出结合双向跨注意力与门控多模态单元（GMU）的融合架构，经50次Optuna搜索优化参数，最终在验证集上取得0.7394的宏平均F1，相对最优单模态基线提升11.0%，证实显式跨模态交互能捕获孤立模态无法提供的互补线索。最终预测基于该模型生成，并按挑战规程提交至官方私有测试集。代码已公开于https://github.com/yassineouzar/IUSD_AH/
原文摘要 · Abstract (English)
We present a multimodal framework for Ambivalence/Hesitancy (A/H) recognition in video, developed for the ABAW11 challenge at ECCV 2026. The proposed approach fuses textual, acoustic, and visual modalities extracted from the BAH dataset using three pretrained encoders: F2LLM-v2-0.6B for transcripts (1024-d), WavLM-Large for audio (1024-d), and VideoMAE V2 for facial video (768-d). We first establish comprehensive unimodal baselines using classical classifiers (MLP, Random Forest, GBDT), each optimized via Optuna, and obtain a best unimodal Macro F1 of \textbf{0.6659} on the test set using text features alone -- substantially outperforming the zero-shot Video-LLaVA baseline (Macro F1: 0.2827). Building on these baselines, we propose a multimodal fusion architecture that combines bidirectional cross-attention across all three modalities with a Gated Multimodal Unit (GMU), with both architectural and optimization hyperparameters selected through a 50-trial Optuna search. This model achieves a Macro F1 of \textbf{0.7394} on the validation set, a relative improvement of 11.0\% over the best unimodal baseline, confirming that explicit cross-modal interaction captures complementary cues that no single modality provides in isolation. Final predictions on the official, unlabeled private test set are generated using this model and submitted according to the challenge protocol. Code is publicly available at https://github.com/yassineouzar/IUSD_AH/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。