通过反向删除推理步骤，检验大模型的思维链是否真实反映其内部认知。
Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps
- 用反向删除关键推理步骤的方法，测试思维链是否忠实于模型参数信念。
- 实验显示，删除特定步骤后，模型预测常发生改变，说明部分思维链是真实的。
- 适合关注模型可解释性与推理可信度的研究者参考。
当被要求逐步思考时，语言模型会产生一段思维链（CoT），即声称用于得出答案的一系列推理步骤。尽管关于思维链提示已有大量研究，但尚不清楚这些输出的推理是否忠实地反映了模型的参数信念。本文提出一种测量参数忠实性的框架，并引入基于反向删除推理步骤（FUR）的方法。FUR 从模型参数中消除推理步骤所包含的信息，并通过观察预测结果的变化来衡量忠实性。在四个语言模型和五个多跳多选问答数据集上的实验表明，FUR 能够通过删除关键步骤精确改变模型对特定样本的预测，揭示了思维链在参数层面的忠实性。进一步分析显示，删除后的模型生成的思维链支持不同答案，暗示反向删除产生了更深层的影响。
原文摘要 · Abstract (English)
When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models' prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。