分析大模型推理过程的因果关系，揭示每一步思考的关键影响。
Local Causal Attribution of Chain-of-Thought Reasoning
- 构建推理步骤间的因果模型，量化每步对后续输出的影响。
- 在5个数据集和4个模型上验证，结果比现有方法更贴近真实行为。
- 适合研究模型可解释性、安全性和不同领域推理差异的人看。
理解语言模型推理过程中的因果结构，对透明性和安全性至关重要。本文采用局部方法，分析特定链式思维轨迹中各个组件（称为单元）之间的因果关系。我们为这些单元构建结构因果模型，并将每个单元与生成后续输出单元的对数概率相关联。提出的算法AttriCoT是一种黑箱方法，通过$O(U)$次前向传播估计因果模型中的重要性参数，其中$U$为单元数量。在5个数据集和4个推理模型上的扰动曲线评估表明，AttriCoT生成的归因结果比其他方法更忠实于模型行为。归因结果还揭示了不同模型和领域间推理结构的显著差异。
原文摘要 · Abstract (English)
Understanding the causal structure of a language model's thought process is a problem of significant importance for both transparency and safety. In this work, we take a local approach toward this goal by analyzing the causal relationships among individual components, termed units, of a given, specific chain-of-thought trace. We construct a structural causal model on these units and relate each unit to the log probability of generating (subsequent) output units. Our algorithm, termed AttriCoT, is a black-box method that performs attribution by estimating importance parameters in the structural causal model using $O(U)$ forward passes through the model, where $U$ is the number of units. Evaluation of perturbation curves across 5 datasets and 4 reasoning models shows that AttriCoT produces attributions that are more faithful to the model's behavior than alternative methods. The attribution results also reveal notable differences in thought structure between models and domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。