发现大模型能从填充符中恢复推理过程，揭示其隐式计算机制
Understanding Hidden Computations in Chain-of-Thought Reasoning
- 通过逐层分析注意力特征，逆向重构被填充的推理步骤
- 替换推理链为'...'符号后模型性能不变，证明隐藏计算存在
- 为提升大模型推理可解释性提供新思路，适合模型透明性研究者
Chain-of-Thought（CoT）提示显著提升了大语言模型的推理能力。然而，近期研究发现，即使将CoT替换为填充符（如'...'），模型仍能完成复杂推理任务，引发对模型内部如何处理和表示推理步骤的疑问。本文研究了在训练时使用填充CoT序列的Transformer模型中解码这些隐藏字符的方法。通过使用logit lens方法分析逐层表示并考察词元排名，我们证明隐藏字符可在不损失性能的情况下被恢复。研究结果揭示了Transformer模型的内在机制，为提升语言模型推理的可解释性与透明性开辟了新路径。
原文摘要 · Abstract (English)
Chain-of-Thought (CoT) prompting has significantly enhanced the reasoning abilities of large language models. However, recent studies have shown that models can still perform complex reasoning tasks even when the CoT is replaced with filler(hidden) characters (e.g., "..."), leaving open questions about how models internally process and represent reasoning steps. In this paper, we investigate methods to decode these hidden characters in transformer models trained with filler CoT sequences. By analyzing layer-wise representations using the logit lens method and examining token rankings, we demonstrate that the hidden characters can be recovered without loss of performance. Our findings provide insights into the internal mechanisms of transformer models and open avenues for improving interpretability and transparency in language model reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。