用思维链提升推理模型的即时强化学习能力，首次给出理论解释。
Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
- 通过线性Transformer建模，证明思维链等价于重复执行时序差分更新。
- 思维链越长，评估误差越小，最终收敛到由上下文长度决定的统计下限。
- 发现理想参数是预训练损失的全局最小值，解释了其在实践中自然出现的原因。
在上下文强化学习（ICRL）中，智能体可在不更新参数的情况下，仅通过额外上下文在推理阶段适应新任务。近期实证研究发现，思维链（CoT）生成能显著增强此能力。本文首次从理论上揭示CoT与ICRL的交互机制。我们在线性Transformer的策略评估设置下进行分析，证明特定参数下的CoT生成过程等价于反复执行时序差分（TD）学习更新。此外，我们提供了有限样本收敛分析，表明策略评估误差随CoT长度几何级下降，最终在由上下文长度决定的统计下限处饱和。我们还证明，所需的理想参数是预训练损失的全局最小值，为这些参数在实践中自发涌现提供了理论依据。
原文摘要 · Abstract (English)
In-context reinforcement learning (ICRL) refers to the ability of RL agents to adapt to new tasks at inference time without parameter updates by conditioning on additional context. Recent empirical studies further demonstrate that Chain-of-Thought (CoT) generation can amplify this ICRL capability. This paper is the first to provide a theoretical understanding on how CoT interacts with ICRL. We conduct our analysis in a policy evaluation setup with linear Transformer. We prove that with specific Transformer parameters, the CoT generation process is equivalent to repeatedly executing temporal difference learning updates. Additionally, we provide finite sample convergence analysis showing that the policy evaluation error decreases geometrically with CoT length and eventually saturates at a statistical floor determined by the context length. We also prove that the desired Transformer parameters are a global minimizer of the pretraining loss, providing a theoretical understanding on the empirical emergence of those parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。