用CLIP融合身体与场景信息，提升自然图像中的情绪识别准确率
A Controlled Study of CLIP-Based Body-Scene Fusion for Emotion Recognition in Context
- 采用双流结构，分别编码人体局部与全图场景特征
- 在EMOTIC数据集上达到34.52% mAP，优于多种改进变体
- 适用于需要理解复杂情境情绪的视觉应用
自然图像中的情绪表现往往不能仅从面部判断，面部可能较小、被遮挡或表情中性，而姿态与场景背景则承载了大量线索。本文在EMOTIC数据集上研究基于图像的上下文感知情绪识别，采用双流模型：使用ResNet-18编码目标人物裁剪区域，用CLIP ViT-B/16编码整幅图像。融合特征同时预测26类情绪标签及连续的愉悦度、唤醒度、支配度。研究对比了简化版因果干预、CLEF-lite去偏方法、ASL调优和类别平衡采样等策略，均未超越原始双流模型。该模型在测试集上取得34.52% mAP。尽管CLIP提供了丰富的场景语义，但简化后的因果、反事实与罕见类别训练方法未能自动提升性能。多数错误集中在稀有与细微情绪类别，提示后续应关注标签间关系与更精细的主体-场景交互建模。
原文摘要 · Abstract (English)
Apparent emotion in natural images is often not visible from the face alone. The face may be small, hidden, or neutral, while posture and scene context carry much of the evidence. This work studies context-aware emotion recognition on EMOTIC with an image-only two-stream model. A ResNet-18 body stream encodes the target-person crop, and a CLIP ViT-B/16 scene stream encodes the full image. The fused feature predicts 26 categorical emotion labels and the continuous valence, arousal, and dominance values. This study examines whether small context-debiasing or rare-class training changes still help after adding a CLIP scene encoder. The clean two-stream model is compared with simplified CCIM-style intervention, CLEF-lite context-bias subtraction, ASL tuning, and class-balanced sampling under the same implementation pipeline. No tested variant improves over the clean two-stream model, which achieves 34.52% mAP on the EMOTIC test split. CLIP gives the model broad scene semantics, but the simplified causal, counterfactual, and rare-class changes do not automatically improve performance. Most remaining errors are in rare and subtle emotion categories, so the next step should focus on label relationships and finer subject-context interaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。