大模型推理时能自发做强化学习，自我改进回答质量。
Reward Is Enough: LLMs Are In-Context Reinforcement Learners
- 用多轮提示让大模型在推理中根据奖励信号优化输出。
- 在数学竞赛和创意写作任务上，性能超越Self-Refine等基线。
- 即使奖励由自身生成，也能持续提升，适合测试时扩展。
强化学习（RL）是解决序列决策问题的框架。本文发现，大型语言模型（LLMs）在推理阶段会自发涌现出类似强化学习的行为，我们称之为上下文内强化学习（ICRL）。为此，我们提出一种简单的多轮提示框架——ICRL提示，用于推理时的自我改进。模型每输出一次，就收到一个数值化反馈（奖励）；下一轮提示中，将之前所有响应及其奖励拼接作为上下文。随着上下文增长，回答质量持续提升，表明模型能在推理过程中优化标量奖励信号，表现出类强化学习行为。我们在24点游戏、创意写作、ScienceWorld及奥数竞赛（AIME和HMMT）上评估，显著优于Self-Refine和Reflexion等基线。值得注意的是，即使奖励由同一模型生成，性能仍可提升，揭示了一种新的测试时扩展范式。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is a framework for solving sequential decision-making problems. In this work, we demonstrate that, surprisingly, RL emerges during the inference time of large language models (LLMs), a phenomenon we term in-context RL (ICRL). To reveal this capability, we introduce a simple multi-round prompting framework, we call ICRL prompting, for inference-time self-improvement. The goal of ICRL prompting is to guide LLMs to perform reinforcement learning during inference for self-improvement on a given task. After each response, the model receives numerical scalar feedback, denoted as a reward. In the next round, we prompt the LLM again together with a context that concatenates all prior responses and their associated rewards. We consistently observe that response quality improves as the context grows. In other words, the LLM can optimize scalar reward signals during inference, exhibiting behavior analogous to reinforcement learning. We evaluate ICRL prompting on Game of 24, creative writing, ScienceWorld, and Olympiad-level math competitions (AIME and HMMT), demonstrating significant improvements over baselines such as Self-Refine and Reflexion. Notably, even when the reward signals are generated by the same LLM, ICRL prompting still improves performance, highlighting a promising new paradigm for test-time scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。