测试大模型推理与答案的分离现象，发现推理正确却答错。
Correct Chains, Wrong Answers: Dissociating Reasoning from Output in LLM Logic
- 设计新测试区分逻辑与名称，隔离真实推理能力
- 深度7时，31个错误均推理正确但答案错误
- 适合研究模型真实推理能力与幻觉机制的学者
大型语言模型可在每一步链式思维推理中完全正确，但仍给出错误最终答案。我们提出新型操作符测试基准，将操作符逻辑与其名称分离，实现对真实推理与模式检索的严格区分。在五种模型上（最多8,100个问题），评估深度1-10下不熟悉名称的布尔操作符表现，揭示现有基准无法检测的推理-输出分离现象。以Claude Sonnet 4为例，在深度7时，全部31个错误均有可验证正确的推理过程但答案错误；混合操作符链中的17/19个错误呈现相同模式。测试发现两类失败：深度2处策略失败（尝试简短检索，较支架提升62个百分点）；深度7处内容失败（推理完整但系统性出错，提升8-30个百分点，干预后0/300错误）。一个特洛伊操作符（用陌生名称表示XOR真值表）证实名称本身不阻碍推理（p ≥ 0.49），而Llama模型在深度8-9时新颖性差距扩大至28个百分点，特洛伊操作符准确率达92-100%，成功将真实逻辑难题与名称陌生度区分开。
原文摘要 · Abstract (English)
LLMs can execute every step of chain-of-thought reasoning correctly and still produce wrong final answers. We introduce the Novel Operator Test, a benchmark that separates operator logic from operator name, enabling rigorous distinction between genuine reasoning and pattern retrieval. By evaluating Boolean operators under unfamiliar names across depths 1-10 on five models (up to 8,100 problems each), we demonstrate a reasoning-output dissociation that existing benchmarks cannot detect. At Claude Sonnet 4's depth 7, all 31 errors have verifiably correct reasoning yet wrong declared answers; 17/19 errors in mixed-operator chains exhibit the same pattern. The benchmark reveals two failure types: strategy failures at depth 2, where models attempt terse retrieval (+62pp from scaffolding), and content failures at depth 7, where models reason fully but err systematically (+8-30pp, 0/300 errors post-intervention). A Trojan operator (XOR's truth table under a novel name) confirms name alone does not gate reasoning (p >= 0.49), while Llama's novelty gap widens to 28pp at depth 8-9 with the Trojan at 92-100%, isolating genuine difficulty with novel logic from name unfamiliarity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。