通过面部动作单元约束推理,让表情识别更可信。
TAG: Thinking with Action Unit Grounding for Facial Expression Recognition
- 用动作单元(AUs)定位视觉线索,强制模型推理与人脸区域对齐。
- 在三个数据集上超越主流视觉语言模型,准确率提升3.2%~5.1%。
- 适合需要可解释、抗幻觉的高可靠性表情识别场景。
表情识别(FER)是细粒度视觉理解任务,可靠预测需基于局部且有意义的面部线索。近期视觉-语言模型(VLMs)能生成自然语言解释,但其推理常缺乏视觉依据,产生流畅却不可验证的理由,易受数据集差异影响而表现不稳定。本文提出TAG(Thinking with Action Unit Grounding),一种显式约束多模态推理与面部动作单元(AUs)对齐的框架。TAG要求中间推理步骤必须基于与AUs相关的面部区域,从而生成带有可验证视觉证据的预测。模型通过在带AU标注的推理轨迹上进行监督微调,并结合感知动作单元的强化学习奖励,使预测区域与外部AU检测器对齐。在RAF-DB、FERPlus和AffectNet三个数据集上,TAG持续优于多个开源与闭源的VLM基线,同时显著提升视觉忠实度。消融实验与偏好测试进一步表明,基于AUs的奖励机制稳定了推理过程,缓解了幻觉问题,证明结构化接地的中间表示对可信多模态推理至关重要。代码将公开于 https://github.com/would1920/FER_TAG。
原文摘要 · Abstract (English)
Facial Expression Recognition (FER) is a fine-grained visual understanding task where reliable predictions require reasoning over localized and meaningful facial cues. Recent vision--language models (VLMs) enable natural language explanations for FER, but their reasoning is often ungrounded, producing fluent yet unverifiable rationales that are weakly tied to visual evidence and prone to hallucination, leading to poor robustness across different datasets. We propose TAG (Thinking with Action Unit Grounding), a vision--language framework that explicitly constrains multimodal reasoning to be supported by facial Action Units (AUs). TAG requires intermediate reasoning steps to be grounded in AU-related facial regions, yielding predictions accompanied by verifiable visual evidence. The model is trained via supervised fine-tuning on AU-grounded reasoning traces followed by reinforcement learning with an AU-aware reward that aligns predicted regions with external AU detectors. Evaluated on RAF-DB, FERPlus, and AffectNet, TAG consistently outperforms strong open-source and closed-source VLM baselines while simultaneously improving visual faithfulness. Ablation and preference studies further show that AU-grounded rewards stabilize reasoning and mitigate hallucination, demonstrating the importance of structured grounded intermediate representations for trustworthy multimodal reasoning in FER. The code will be available at https://github.com/would1920/FER_TAG .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。