提出因果启发的评估框架，精准识别大模型推理中的虚假步骤。
FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning
- 用可控扰动分离真实因果与偏见干扰，提升推理链内部一致性评估
- 在GSM8K等3个数据集上显著提升可信推理轨迹选择效果
- 适合需要可靠推理验证的研究者与应用开发者
链式思维（CoT）提示提升了大模型的推理能力，但模型常生成看似连贯却包含不忠实中间步骤的解释。现有自评估方法易受固有偏见影响：模型可能自信地认可逻辑连贯性，即使步骤间推导无效，导致不可靠的忠实度评估。本文提出FACT-E，一种基于因果启发的推理质量评估框架。FACT-E利用可控扰动作为工具变量，分离真实步骤间依赖关系与由偏见驱动的伪相关，从而获得更可靠的内部一致性评估（即“链内忠实度”）。为筛选可信推理路径，FACT-E联合考虑“链内忠实度”与“推理链到答案的一致性”，确保所选链条既内部忠实又支持正确答案。在GSM8K、MATH和CommonsenseQA上的实验表明，FACT-E显著提升了推理轨迹选择性能，并生成更强的上下文学习范例。该方法在噪声环境下仍能可靠检测错误推理，提供了一种鲁棒的可信大模型推理评估指标。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting has improved LLM reasoning, but models often generate explanations that appear coherent while containing unfaithful intermediate steps. Existing self-evaluation approaches are prone to inherent biases: the model may confidently endorse coherence even when the step-to-step implication is not valid, leading to unreliable faithfulness evaluation. We propose FACT-E, a causality-inspired framework for evaluating CoT quality. FACT-E uses controlled perturbations as an instrumental signal to separate genuine step-to-step dependence from bias-driven artifacts, producing more reliable faithfulness estimates (\textit{intra-chain faithfulness}). To select trustworthy trajectories, FACT-E jointly considers \textit{intra-chain faithfulness} and \textit{CoT-to-answer consistency}, ensuring that selected chains are both faithful internally and supportive of the correct final answer. Experiments on GSM8K, MATH, and CommonsenseQA show that FACT-E improves reasoning-trajectory selection and yields stronger in-context learning exemplars. FACT-E also reliably detects flawed reasoning under noisy conditions, providing a robust metric for trustworthy LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。