arXiv:2511.11126cs.CLcs.CV2025-11

用大模型增强 meme 文本,双阶段融合提升情感理解准确率

Enhancing Meme Emotion Understanding with Multi-Level Modality Enhancement and Dual-Stage Modal Fusion

  • 用 MLLM 逐步挖掘 meme 隐含语义,增强原始文本信息
  • 分两阶段融合:先浅层结合原始图文,再深度融合增强特征
  • 在两个数据集上分别提升 4.3% 和 3.4% 的 F1 得分,适合情感分析研究者

随着社交媒体和网络文化的兴起,meme 成为表达情绪倾向的重要媒介,推动了 meme 情感理解(MEU)的研究。现有方法仍面临两大挑战:缺乏细粒度的多模态融合策略,以及对 meme 隐含意义和背景知识挖掘不足。为此,我们提出 MemoDetector 框架。首先引入四步文本增强模块,利用多模态大模型(MLLM)的能力,逐步推断并提取 meme 的隐含与上下文信息,显著丰富原始内容,并为下游分类提供指导。其次设计双阶段模态融合策略:第一阶段对原始图像与文本进行浅层融合,第二阶段深度整合增强后的视觉与文本特征。该分层融合机制更有效捕捉跨模态情感线索。在 MET-MEME 与 MOOD 两个数据集上的实验表明,该方法持续优于当前最优基线,F1 分别提升 4.3% 与 3.4%。消融实验与深入分析验证了方法的有效性与鲁棒性,展现出在 MEU 领域的强大潜力。代码已公开于 https://github.com/singing-cat/MemoDetector。

原文摘要 · Abstract (English)

With the rapid rise of social media and Internet culture, memes have become a popular medium for expressing emotional tendencies. This has sparked growing interest in Meme Emotion Understanding (MEU), which aims to classify the emotional intent behind memes by leveraging their multimodal contents. While existing efforts have achieved promising results, two major challenges remain: (1) a lack of fine-grained multimodal fusion strategies, and (2) insufficient mining of memes' implicit meanings and background knowledge. To address these challenges, we propose MemoDetector, a novel framework for advancing MEU. First, we introduce a four-step textual enhancement module that utilizes the rich knowledge and reasoning capabilities of Multimodal Large Language Models (MLLMs) to progressively infer and extract implicit and contextual insights from memes. These enhanced texts significantly enrich the original meme contents and provide valuable guidance for downstream classification. Next, we design a dual-stage modal fusion strategy: the first stage performs shallow fusion on raw meme image and text, while the second stage deeply integrates the enhanced visual and textual features. This hierarchical fusion enables the model to better capture nuanced cross-modal emotional cues. Experiments on two datasets, MET-MEME and MOOD, demonstrate that our method consistently outperforms state-of-the-art baselines. Specifically, MemoDetector improves F1 scores by 4.3\% on MET-MEME and 3.4\% on MOOD. Further ablation studies and in-depth analyses validate the effectiveness and robustness of our approach, highlighting its strong potential for advancing MEU. Our code is available at https://github.com/singing-cat/MemoDetector.

情感理解多模态meme大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。