构建幽默多模态隐喻数据集，测试大模型理解能力
Hummus: A Dataset of Humorous Multimodal Metaphor Use
- 基于隐喻与幽默理论设计新标注方案
- 1000组图文对中专家标注幽默隐喻用法
- 揭示当前多模态大模型理解困难点
隐喻与幽默有诸多共通之处，隐喻是常见幽默机制。本研究聚焦多模态隐喻的幽默性，此前未受充分关注。借鉴幽默的不协调理论、概念隐喻理论及阿姆斯特丹自由大学隐喻语料库的标注框架，我们为图像-标题对中的幽默多模态隐喻使用开发了新的标注方案。构建了名为Hummus的数据集，包含从《纽约客》标题竞赛语料库中抽取的1000组图像-标题对，并由专家进行标注。利用该数据集，我们测试了当前先进的多模态大语言模型（MLLMs）在检测和理解幽默多模态隐喻方面的能力。实验表明，现有MLLMs在整合视觉与文本信息以理解幽默多模态隐喻方面仍存在显著困难。数据集与代码已开源至github.com/xiaoyuisrain/humorous-multimodal-metaphor-use。
原文摘要 · Abstract (English)
Metaphor and humor share a lot of common ground, and metaphor is one of the most common humorous mechanisms. This study focuses on the humorous capacity of multimodal metaphors, which has not received due attention in the community. We take inspiration from the Incongruity Theory of humor, the Conceptual Metaphor Theory, and the annotation scheme behind the VU Amsterdam Metaphor Corpus, and developed a novel annotation scheme for humorous multimodal metaphor use in image-caption pairs. We create the Hummus Dataset of Humorous Multimodal Metaphor Use, providing expert annotation on 1k image-caption pairs sampled from the New Yorker Caption Contest corpus. Using the dataset, we test state-of-the-art multimodal large language models (MLLMs) on their ability to detect and understand humorous multimodal metaphor use. Our experiments show that current MLLMs still struggle with processing humorous multimodal metaphors, particularly with regard to integrating visual and textual information. We release our dataset and code at github.com/xiaoyuisrain/humorous-multimodal-metaphor-use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。