提出新评估方法，检测大模型逻辑推理中虚假连贯性
Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning
- 设计双查询评估框架，同时衡量一致性与明确判断
- 1.1B模型覆盖率79.4%但全错，3B模型仅7.4%覆盖却无矛盾
- 适合关注模型真实推理能力的研究者和开发者
在知识密集型领域部署的大语言模型进行逻辑推理时，存在一种微妙但关键的失效：通过系统性回避表态可实现形式上的连贯性。模型若不对蕴含或反驳做出承诺，虽满足否定一致性，实则毫无实用价值。本文提出「承诺下的连贯性」（CUC）评估范式，联合测量一致性与决断力。创新包括：(1) 承诺分数 $c(φ) = p(φ) + p( egφ)$，量化分配给明确结论的概率质量；(2) 通过归一化是/否对数概率的确定性诱导协议，消除采样方差；(3) 三分类决策框架（真/假/不确定），将连贯性-承诺权衡转化为可度量指标。在204个FOLIO示例上对四个开源小模型（1B–3B）的实验揭示显著分界线：Qwen2.5-3B的期望否定矛盾率仅0.025，但覆盖率仅7.4%；TinyLlama-1.1B覆盖率79.4%，但每例均出错。仅评估连贯性的方法会优先排名回避表态的模型；而CUC揭示其为虚假连贯。该分界线在LogiQA~v2上也成立（ρ=0.97）。我们主张评估应同时报告连贯性与非空洞承诺，并发布标准化评估工具包。
原文摘要 · Abstract (English)
Large language models (LLMs) deployed for logical reasoning in knowledge-intensive domains exhibit a subtle but critical failure: coherence can be vacuously achieved through systematic abstention. A model that withholds commitment to either entailment or refutation satisfies negation consistency while providing no utility. We introduce Coherence Under Commitment (CUC), a dual-query evaluation paradigm that jointly measures consistency and decisiveness. CUC contributes three innovations: (1) a commitment score $c(φ) = p(φ) + p(\lnotφ)$ quantifying probability mass allocated to decisive outcomes; (2) a \textbf{deterministic elicitation protocol} via normalized YES/NO log probabilities, eliminating sampling variance; and (3) a 3-way decision framework (True/False/Uncertain) operationalizing the coherence-commitment trade-off into metrics. Experiments on four open-weight LLMs (1B-3B) across 204 FOLIO examples expose a sharp frontier. Qwen2.5-3B achieves near-zero contradiction ($\mathbb{E}[v_{\mathrm{neg}}]{=}0.025$) but only $7.4\%$ coverage, while TinyLlama-1.1B reaches $79.4\%$ coverage with violations on every example. Coherence-only evaluation would rank the abstaining model first; CUC exposes this as vacuous, and the frontier generalizes to LogiQA~v2 ($ρ{=}0.97$). We argue that evaluation must report both coherence and non-vacuous commitment and release a toolkit for standardized assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。