测试发现大模型难懂政治话语中的隐含意思。
They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discourse
- 用意大利政治演讲数据集测试模型理解隐含语义能力
- 多模型在预设和言外之意任务中表现不佳
- 揭示当前大模型在政治话语理解上的根本局限
隐含内容在政治话语中起关键作用，说话者系统性地运用会话含义和预设等语用策略影响听众。尽管大语言模型（LLMs）在复杂语义与语用理解任务中表现出色，显示出识别和解释隐含内容的潜力，但其在政治话语中的应用仍鲜有研究。本文首次利用大规模IMPAQTS语料库——包含标注了操纵性隐含内容的意大利政治演讲——提出方法来评估LLMs在该挑战性任务中的有效性。通过多项选择题和开放式生成任务，我们发现所有测试模型均难以准确解读预设和言外之意。结论表明，当前大模型缺乏准确理解高度隐含语言（如政治话语）所必需的关键语用能力。同时，我们指出有前景的趋势和未来改进方向。数据与代码已开源：https://github.com/WalterPaci/IMPAQTS-PID。
原文摘要 · Abstract (English)
Implicit content plays a crucial role in political discourse, where speakers systematically employ pragmatic strategies such as implicatures and presuppositions to influence their audiences. Large Language Models (LLMs) have demonstrated strong performance in tasks requiring complex semantic and pragmatic understanding, highlighting their potential for detecting and explaining the meaning of implicit content. However, their ability to do this within political discourse remains largely underexplored. Leveraging, for the first time, the large IMPAQTS corpus, which comprises Italian political speeches with the annotation of manipulative implicit content, we propose methods to test the effectiveness of LLMs in this challenging problem. Through a multiple-choice task and an open-ended generation task, we demonstrate that all tested models struggle to interpret presuppositions and implicatures. We conclude that current LLMs lack the key pragmatic capabilities necessary for accurately interpreting highly implicit language, such as that found in political discourse. At the same time, we highlight promising trends and future directions for enhancing model performance. We release our data and code at https://github.com/WalterPaci/IMPAQTS-PID
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。