发现大模型在上下文学习中自发出现强化学习的时序差分信号
Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models
- 用稀疏自编码器分析模型中间表示，发现与时序差分误差高度匹配的特征
- 模型仅训练预测下一个词，却在上下文学习中显现出解决强化学习问题的能力
- 通过干预实验验证这些特征对决策计算具有因果作用，适合研究机制的AI学者
上下文学习是大型语言模型（LLMs）基于输入提示中的少量示例进行适应的普遍能力。随着LLMs上下文学习能力不断提升，理解其内在机制变得愈发重要。特别是，目前尚不清楚LLMs如何在上下文学习中解决特定类型问题，如强化学习（RL）问题。通过三个不同任务，我们首先证明了Llama 3 70B能够在上下文学习中解决简单强化学习问题。随后，我们利用稀疏自编码器（SAEs）分析Llama的残差流，发现了与时序差分（TD）误差高度匹配的表示。值得注意的是，这些表示是在模型仅以预测下一个词为目标训练的前提下出现的。通过精心设计的干预实验，我们验证了这些表示确实对TD误差和Q值的计算具有因果影响。综上所述，本工作建立了一种利用SAEs研究和操控上下文学习的系统方法，为更深入的机制理解铺平道路。
原文摘要 · Abstract (English)
In-context learning, the ability to adapt based on a few examples in the input prompt, is a ubiquitous feature of large language models (LLMs). However, as LLMs' in-context learning abilities continue to improve, understanding this phenomenon mechanistically becomes increasingly important. In particular, it is not well-understood how LLMs learn to solve specific classes of problems, such as reinforcement learning (RL) problems, in-context. Through three different tasks, we first show that Llama $3$ $70$B can solve simple RL problems in-context. We then analyze the residual stream of Llama using Sparse Autoencoders (SAEs) and find representations that closely match temporal difference (TD) errors. Notably, these representations emerge despite the model only being trained to predict the next token. We verify that these representations are indeed causally involved in the computation of TD errors and $Q$-values by performing carefully designed interventions on them. Taken together, our work establishes a methodology for studying and manipulating in-context learning with SAEs, paving the way for a more mechanistic understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。