用因果图思维提升多模态幽默理解能力
CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension

- 将幽默背后的因果与上下文关系建模为轻量图结构
- 在4个数据集上提升幽默理解1-20%、检测1-3%
- 适合研究幽默理解、多模态推理的学者与工程师
大规模视觉语言模型(VLMs)在多种多模态任务中表现出色,但理解幽默仍具挑战性,因其依赖图像与文本模态间实体、事件、语境及隐含关系的细微互动。这些互动常涉及复杂推理链,传统提示或线性思维链难以捕捉。本文提出CaRGo-T(因果推理思维图),将多模态幽默背后的因果与上下文关系表示为轻量级图结构。该图由VLM生成代码化表示,可被同一或另一VLM在零样本或上下文学习设置下解析并输出最终预测。我们在涵盖讽刺、反语和梗图等多样喜剧内容的4个数据集上评估了CaRGo-T,实验表明其在主流商业与开源VLM上均优于现有基于推理的基线,幽默理解提升约1-20%,幽默检测提升1-3%。进一步互信息分析显示,CaRGo-T生成的推理表示包含更相关的目标输出信息。代码已开源。
原文摘要 · Abstract (English)
Large-scale vision-language models (VLMs) have demonstrated remarkable versatility across a wide range of multimodal tasks. However, understanding humor remains challenging because humorous content often depends on subtle interactions among entities, events, context, and implicit relationships across image and text modalities. These interactions can involve complex chains of reasoning that are difficult to capture through conventional prompting or linear chain-of-thought reasoning. In this work, we propose CaRGo-T (Causal Reasoning Graph-of-Thought), a reasoning framework that represents the causal and contextual relationships underlying multimodal humor as a lightweight graph-based reasoning structure. The graph is serialized into a code-based representation generated by a VLM, which can subsequently be interpreted by the same or a different VLM to produce the final prediction in zero-shot or in-context learning settings. We evaluate CaRGo-T on humor understanding and humor detection across four datasets spanning diverse forms of comedic content, including satire, sarcasm, and memes. Experiments with state-of-the-art commercial and open-source VLMs show that CaRGo-T consistently improves performance over existing reasoning-based baselines, achieving gains of approximately 1-20% on humor understanding and 1-3% on humor detection. Further analysis using mutual information indicates that the reasoning representations produced by CaRGo-T contain more information relevant to the target output than those generated by baseline reasoning approaches. Code is available at https://github.com/abhi1nandy2/CaRGo-T.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。