用简单反馈'再试一次'就能让大模型学会多轮推理与纠错。
A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning
- 用单一反馈'再试一次'设计多轮强化学习训练方法。
- 多轮推理准确率提升最高达14%，且单轮性能不变。
- 适合想提升模型反馈响应能力的研究者与工程师。
多轮问题求解对大型推理模型（LRMs）而言至关重要，但极具挑战性，因为模型需反思自身推理并根据上下文反馈进行修正。现有强化学习（RL）方法在单轮可验证奖励范式下训练大模型，但我们发现此类训练会使模型丧失跨多轮解决问题的能力，难以基于反馈修正答案，导致重复输出。我们提出：大模型能否在多轮情境中学会自我反思？本工作发现，仅在错误答案后使用简单的一元反馈（如“再试一次”）进行多轮强化学习训练，既能保持单轮性能，又能显著提升多轮推理能力。我们提出“一元反馈作为观测”（UFO）机制，利用迭代求解中常见的最小化用户反馈，可无缝集成至现有单轮RL训练流程。实验表明，采用UFO的强化学习训练在保持单轮性能的同时，使多轮推理准确率最高提升14%，显著增强语言模型对多轮反馈的响应能力。为进一步减少正确答案所需轮数并鼓励出错时产生多样化推理，我们设计了引导模型每轮生成谨慎、深思熟虑回答的奖励结构。
原文摘要 · Abstract (English)
Multi-turn problem solving is critical yet challenging for Large Reasoning Models (LRMs) to reflect on their reasoning and revise from feedback. Existing Reinforcement Learning (RL) methods train large reasoning models on a single-turn paradigm with verifiable rewards. However, we observe that models trained with existing RL paradigms often lose their ability to solve problems across multiple turns and struggle to revise answers based on contextual feedback, leading to repetitive responses. We ask: can LRMs learn to reflect their answers in a multi-turn context? In this work, we find that training models with multi-turn RL using only unary feedback (e.g., "Let's try again") after wrong answers can improve both single-turn performance and multi-turn reasoning. We introduce Unary Feedback as Observation (UFO) for reinforcement learning, which uses minimal yet common unary user feedback during iterative problem solving. It can be easily applied to existing single-turn RL training setups. Experimental results show that RL training with UFO keeps single-turn performance and improves multi-turn reasoning accuracy by up to 14%, enabling language models to better react to feedback in multi-turn problem solving. To further minimize the number of turns needed for a correct answer while encouraging diverse reasoning when mistakes occur, we design reward structures that guide models to produce careful and deliberate answers in each turn. Code: https://github.com/lichengliu03/unary-feedback
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。