通过引入上下文检索提升大模型讽刺识别能力,效果显著。
Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection
- 结合外部检索与模型自知识,增强上下文理解
- 在印尼推特数据集上宏F1提升9.87%
- 适合处理文化特定表达的讽刺检测任务
尽管神经网络方法取得进展,讽刺检测仍是自然语言处理中的难题。当前主流依赖预训练语言模型(PLMs)和大语言模型(LLMs),但讽刺文本的复杂性、语言多样性及文化差异使其难以准确识别,且模型常对需额外语境支撑的词项判断不可靠。本文基于先进的语用元认知提示(PMP)方法,提出一种上下文感知的检索增强策略:一方面利用网络检索补充非参数化背景知识,另一方面激发模型自身内部知识以实现自我认知。在Twitter Indonesia Sarcastic、SemEval-2018 Task 3和MUStARD三个数据集上的实验表明,非参数检索使印尼数据集宏F1提升9.87%,自知识检索在SemEval和MUStARD上分别提升3.29%和4.08%。结果凸显了上下文信息对提升大模型讽刺识别性能的重要性,尤其在处理文化特有俚语、引用或模型未见过术语时。代码已公开于https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base。
原文摘要 · Abstract (English)
Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 Task 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. These findings highlight the importance of context in enhancing LLMs performance in sarcasm detection task, particularly the involvement of culturally specific slang, references, or unknown terms to the LLMs. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. The experiment code is available at: https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。