用上下文定义情绪标签，提升视觉情绪识别准确率
EmoGist: Efficient In-Context Learning for Visual Emotion Understanding
- 根据图像聚类生成情绪描述，测试时按相似度选取匹配描述
- 在多标签Memotion数据集上微调F1最高提升12点，单类FI数据集提升8点
- 无需训练，适合快速部署到现有视觉语言模型中
本文提出EmoGist，一种无需训练的上下文学习方法，用于基于大型视觉语言模型（LVLM）进行视觉情绪分类。其核心思路是：情绪标签的语义应随上下文动态调整，因为情绪在图像中的表现高度依赖具体情境。EmoGist通过分析每个情绪类别下示例图像的聚类，预先生成多个情绪标签的描述。测试时，根据测试图像与各聚类中心的余弦相似度，检索最匹配的描述，并与测试图像一同输入快速的LVLM进行分类。实验表明，EmoGist在多标签Memotion数据集上使微平均F1提升最多达12个百分点，在多分类FI数据集上宏平均F1提升最多达8个百分点。
原文摘要 · Abstract (English)
In this paper, we introduce EmoGist, a training-free, in-context learning method for performing visual emotion classification with LVLMs. The key intuition of our approach is that context-dependent definition of emotion labels could allow more accurate predictions of emotions, as the ways in which emotions manifest within images are highly context dependent and nuanced. EmoGist pre-generates multiple descriptions of emotion labels, by analyzing the clusters of example images belonging to each label. At test time, we retrieve a version of description based on the cosine similarity of test image to cluster centroids, and feed it together with the test image to a fast LVLM for classification. Through our experiments, we show that EmoGist allows up to 12 points improvement in micro F1 scores with the multi-label Memotion dataset, and up to 8 points in macro F1 in the multi-class FI dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。