arXiv:2604.15280cs.CVcs.AI2026-04

VLMs难识情绪,因数据偏倚和无法捕捉微表情时间动态。

Why Do Vision Language Models Struggle To Recognize Human Emotions?

论文配图:Why Do Vision Language Models Struggle To Recognize Human Emotions?
图 1 · 摘自论文原文
  • 用稀疏采样忽略关键帧间信息,导致情绪识别失效。
  • 长尾数据使罕见情绪被误归入常见类别,准确率下降15%以上。
  • 提出语言摘要增强上下文,帮助模型理解情绪演变过程。

理解情绪是智能系统与人类交互的基础能力。尽管视觉语言模型(VLMs)在众多视觉任务中取得显著进展,可能为情绪理解提供解决方案,但令人惊讶的是,即使最先进的VLMs在识别人类情绪方面仍表现不佳,甚至不如专用的视觉分类器。本文探讨‘为何VLMs难以识别情绪’,发现面部表情识别(DFER)这一固有的连续动态任务暴露了VLMs的两大关键弱点。首先,情绪数据集天然长尾,而用于预训练的网络规模数据加剧了头部类别偏差,导致稀有情绪被系统性地合并到常见类别中。我们提出了替代采样策略以避免对常见概念的偏好。其次,时间信息对情绪理解至关重要,但VLMs受限于上下文长度和内存中的令牌数量,无法有效表示密集帧序列的时间信息,这对情绪识别构成明显挑战。我们发现,VLMs采用的稀疏时间采样策略与微表情(0.25–0.5秒)的短暂特性本质不匹配,而微表情往往是最重要的情感信号。作为诊断工具,我们提出一种多阶段上下文增强策略:先将‘帧间’信息转化为自然语言摘要,再将该丰富文本上下文与稀疏关键帧一同输入VLM,从而在防止注意力稀释的同时保留情绪发展轨迹。

原文摘要 · Abstract (English)

Understanding emotions is a fundamental ability for intelligent systems to be able to interact with humans. Vision-language models (VLMs) have made tremendous progress in the last few years for many visual tasks, potentially offering a promising solution for understanding emotions. However, it is surprising that even the most sophisticated contemporary VLMs struggle to recognize human emotions or to outperform even specialized vision-only classifiers. In this paper we ask the question 'Why do VLMs struggle to recognize human emotions?', and observe that the inherently continuous and dynamic task of facial expression recognition (DFER) exposes two critical VLM vulnerabilities. First, emotion datasets are naturally long-tailed, and the web-scale data used to pre-train VLMs exacerbates this head-class bias, causing them to systematically collapse rare, under-represented emotions into common categories. We propose alternative sampling strategies that prevent favoring common concepts. Second, temporal information is critical for understanding emotions. However, VLMs are unable to represent temporal information over dense frame sequences, as they are limited by context size and the number of tokens that can fit in memory, which poses a clear challenge for emotion recognition. We demonstrate that the sparse temporal sampling strategy used in VLMs is inherently misaligned with the fleeting nature of micro-expressions (0.25-0.5 seconds), which are often the most critical affective signal. As a diagnostic probe, we propose a multi-stage context enrichment strategy that utilizes the information from 'in-between' frames by first converting them into natural language summaries. This enriched textual context is provided as input to the VLM alongside sparse keyframes, preventing attentional dilution from excessive visual data while preserving the emotional trajectory.

情绪识别视觉语言模型时间建模数据偏倚

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。