构建多语言讽刺检测数据集,含用户互动图谱提升上下文理解。
FanChuan: A Multilingual and Graph-Structured Benchmark For Parody Detection and Analysis
- 基于中英文语料构建7个讽刺数据集,含1.4万用户与2.1万条评论。
- 用户互动图谱显著提升讽刺识别效果,传统方法在某些场景优于大模型。
- 适合研究社交媒体讽刺、文化表达与模型鲁棒性的学者使用。
讽刺是社交媒体中的新兴现象,个体模仿与其立场相反的角色,常用于幽默、挑衅或引发争议。讽刺的检测与分析依赖上下文,对理解文化价值、推动亚文化发展和促进自我表达至关重要。然而,现有研究受限于数据稀缺与多样性不足。为此,我们从英中文语料中构建了7个讽刺数据集,共包含14,755名标注用户和21,210条标注评论。为提供充分上下文,我们收集回复并构建用户互动图谱,弥补现有数据集在上下文信息上的缺失。基于这些数据集,我们在三个关键任务上测试了传统方法与大语言模型(LLMs):(1)讽刺检测,(2)含讽刺的评论情感分析,(3)含讽刺的用户情感分析。大量实验表明,讽刺相关任务对所有模型仍具挑战性,上下文信息至关重要。有趣的是,在某些场景下,结合简单分类器的传统句向量方法反而优于DeepSeek-R1和GPT-o3等先进大模型,凸显讽刺对大模型构成重大挑战。
原文摘要 · Abstract (English)
Parody is an emerging phenomenon on social media, where individuals imitate a role or position opposite to their own, often for humor, provocation, or controversy. Detecting and analyzing parody can be challenging and is often reliant on context, yet it plays a crucial role in understanding cultural values, promoting subcultures, and enhancing self-expression. However, the study of parody is hindered by limited available data and deficient diversity in current datasets. To bridge this gap, we built seven parody datasets from both English and Chinese corpora, with 14,755 annotated users and 21,210 annotated comments in total. To provide sufficient context information, we also collect replies and construct user-interaction graphs to provide richer contextual information, which is lacking in existing datasets. With these datasets, we test traditional methods and Large Language Models (LLMs) on three key tasks: (1) parody detection, (2) comment sentiment analysis with parody, and (3) user sentiment analysis with parody. Our extensive experiments reveal that parody-related tasks still remain challenging for all models, and contextual information plays a critical role. Interestingly, we find that, in certain scenarios, traditional sentence embedding methods combined with simple classifiers can outperform advanced LLMs, i.e. DeepSeek-R1 and GPT-o3, highlighting parody as a significant challenge for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。