评测大模型对对话隐含意义的理解能力,发现推理时表现远差于生成时。
DRInQ: Evaluating Conversational Implicature with Controlled Context Variation

- 设计新基准DRinQ,固定问题表面形式,系统变化上下文以隔离语用推理。
- 顶尖模型生成时能造合理情境,但推理时难以还原真实隐含意义。
- 小模型用结构化提示可提升与人类判断的一致性,适合研究语用推理机制。
人类对话高度依赖语用隐含意义,即说话人传达的并非字面含义。尽管当前大语言模型具备强对话流畅性,但在依赖社会与上下文线索整合推理时仍不可靠,此类推理在文本中很少明确表达。我们提出DRinQ,一个用于评估问题表述中语用隐含意义推理的基准,通过保持问题表面形式不变,系统性地改变上下文以隔离语用变异。为支持可扩展评估,我们设计半自动化流水线,生成带有系统变化的问答-上下文-解释实例。评估显示存在一致的生成-推理不对称:先进模型在引导下可生成合理语用场景,但在推理阶段常无法恢复预期隐含意义。较小模型通过结构化提示能更接近人类判断。对比写作研究进一步揭示互补优势:人类作者倾向于生成安全、可预测的上下文,而模型生成多样化情境,其解释有时超出上下文支持。这些发现凸显建模对话隐含意义的持续挑战,并推动更具上下文敏感性的评估框架发展。
原文摘要 · Abstract (English)
Human conversation relies heavily on conversational implicature, in which speakers convey meanings that are suggested rather than explicitly stated. Although recent large language models exhibit strong conversational fluency, they remain unreliable when interpretation depends on reasoning that integrates social and contextual cues, a process rarely articulated in text. We introduce DRinQ, a benchmark for evaluating pragmatic reasoning about conversational implicature in question utterances, designed to isolate pragmatic variation while holding each question's surface form fixed. To support scalable evaluation, we propose a semi-automated pipeline that produces question-context-interpretation instances with systematic variation. Across evaluations, we find a consistent generation-inference asymmetry: while state-of-the-art models can generate plausible pragmatic scenarios when guided, they often fail to recover the intended implication at inference time. For smaller models, structured prompting improves alignment with human judgments. A comparative writing study further reveals complementary strengths: human authors tend to produce safer, predictable contexts, whereas models generate varied scenarios with interpretations that sometimes exceed contextual support. These findings highlight persistent challenges in modeling conversational implicature and motivate more context-sensitive evaluation frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。