提出评估对话中策略性语言的新框架,揭示大模型在对抗语境下的理解短板。
Strategic Dialogue Assessment: The Crooked Path to Innocence
- 基于格赖斯与博弈论构建对话策略评估框架
- 发现模型规模提升但推理能力反而损害表现
- 适合研究对话策略、法庭质询或可信生成的学者
语言常被用于策略性表达,尤其在高风险对抗场景中,但现有关于语用学与大模型的研究多聚焦合作性,忽视了对抗语境下的策略沟通。为此,我们提出SDA(Strategic Dialogue Assessment)框架,基于格赖斯与博弈论语用学,将ME Game裁判函数改造为可实证估算的形式,以分析对话中的策略行为。该方法引入基于承诺的对话动作分类体系,更精细刻画策略影响;并采用基于格赖斯准则的可估算代理指标,实现如可信度等抽象概念的操作化。通过将话语视为承诺的策略管理,系统评估对话行为对语境目标的推进或破坏作用。我们进一步推导出三个可解释指标:每回合收益(BAT)、每回合惩罚(PAT)与归一化相对收益(NRBAT),量化对话策略效果。同时构建了真实法庭交叉质询的标注数据集CPD,用于验证框架有效性。实验评估多种大模型后发现,尽管模型规模增大,性能有所提升,但推理能力并未带来帮助,反而导致过度复杂化和内部混乱。
原文摘要 · Abstract (English)
Language is often used strategically, particularly in high-stakes, adversarial settings, yet most work on pragmatics and LLMs centers on cooperativity. This leaves a gap in the systematic understanding of strategic communication in adversarial settings. To address this, we introduce SDA (Strategic Dialogue Assessment), a framework grounded in Gricean and game-theoretic pragmatics to assess strategic use of language. It adapts the ME Game jury function to make it empirically estimable for analyzing dialogue. Our approach incorporates two key adaptations: a commitment-based taxonomy of discourse moves, which provides a finer-grained account of strategic effects, and the use of estimable proxies grounded in Gricean maxims to operationalize abstract constructs such as credibility. Together, these adaptations build on discourse theory by treating discourse as the strategic management of commitments, enabling systematic evaluation of how conversational moves advance or undermine discourse goals. We further derive three interpretable metrics-Benefit at Turn (BAT), Penalty at Turn (PAT), and Normalized Relative Benefit at Turn (NRBAT)-to quantify the perceived strategic effects of discourse moves. We also present CPD (the Crooked Path Dataset), an annotated dataset of real courtroom cross-examinations, to demonstrate the framework's effectiveness. Using these tools, we evaluate a range of LLMs and show that LLMs generally exhibit limited pragmatic understanding of strategic language. While model size shows an increase in performance on our metrics, reasoning ability does not help and largely hurts, introducing overcomplication and internal confusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。