解决真实场景下表情识别难题，多模态融合更鲁棒
Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout
- 双分支Transformer用安全交叉注意力动态融合视觉音频
- 在缺模态时靠音频预测，提升复杂场景适应力
- 适合处理数据不均衡与遮挡问题的工业级应用
真实环境中的情绪识别受部分遮挡、模态缺失和严重类别不平衡影响。针对此问题，我们为情感行为分析在野挑战（ABAW）提出一种多模态框架，动态融合视觉与音频表征。采用双分支Transformer架构，结合安全交叉注意力机制与模态丢弃策略，使网络在视觉信息缺失时可依赖音频预测。为缓解Aff-Wild2数据集的长尾分布，使用焦点损失优化，并引入滑动窗口软投票策略捕捉动态情绪变化，减少帧级别分类抖动。实验表明，该框架有效应对模态缺失与时空依赖复杂性，在Aff-Wild2验证集上达到60.79%准确率和0.5029的F1分数。
原文摘要 · Abstract (English)
Emotion recognition in real-world environments is hindered by partial occlusions, missing modalities, and severe class imbalance. To address these issues, particularly for the Affective Behavior Analysis in-the-wild (ABAW) Expression challenge, we propose a multimodal framework that dynamically fuses visual and audio representations. Our approach uses a dual-branch Transformer architecture featuring a safe cross-attention mechanism and a modality dropout strategy. This design allows the network to rely on audio-based predictions when visual cues are absent. To mitigate the long-tail distribution of the Aff-Wild2 dataset, we apply focal loss optimization, combined with a sliding-window soft voting strategy to capture dynamic emotional transitions and reduce frame-level classification jitter. Experiments demonstrate that our framework effectively handles missing modalities and complex spatiotemporal dependencies, achieving an accuracy of 60.79% and an F1-score of 0.5029 on the Aff-Wild2 validation set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。