揭秘大模型推理时的思维链机制，发现其如何激活更广泛知识。
Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation
- 分析思维链触发时的解码、投影与激活过程，揭示内部变化。
- 生成过程中词元概率分布更集中，最终层神经元激活范围更广。
- 适合研究模型推理机制或优化提示工程的研究者阅读。
思维链提示显著提升了大语言模型的推理能力，但其内在机制仍不明确。本文从解码、投影和激活三个关键维度出发，深入探究使用思维链时模型内部的变化。研究发现，大模型在生成过程中能有效模仿示例格式，并将其与问题理解相结合；虽然生成时词元概率存在波动，但最终输出的概率分布更加集中；同时，模型在最终层激活了更广泛的神经元，表明其进行了更全面的知识检索。相关代码与数据将在论文录用后公开。
原文摘要 · Abstract (English)
Chain-of-Thought prompting has significantly enhanced the reasoning capabilities of large language models, with numerous studies exploring factors influencing its performance. However, the underlying mechanisms remain poorly understood. To further demystify the operational principles, this work examines three key aspects: decoding, projection, and activation, aiming to elucidate the changes that occur within models when employing Chainof-Thought. Our findings reveal that LLMs effectively imitate exemplar formats while integrating them with their understanding of the question, exhibiting fluctuations in token logits during generation but ultimately producing a more concentrated logits distribution, and activating a broader set of neurons in the final layers, indicating more extensive knowledge retrieval compared to standard prompts. Our code and data will be publicly avialable when the paper is accepted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。