用强化学习让对话模型边做动作边思考,提升推理与执行能力。
When Actions Teach You to Think: Reasoning-Action Synergy via Reinforcement Learning in Conversational Agents
- 通过强化学习让模型从任务结果中自主学习推理与工具调用策略。
- 相比监督微调模型提升1.5%准确率,较基础模型提高40%性能。
- 适合需要高可靠性和泛化能力的智能对话系统研发人员。
监督微调(SFT)是提升大语言模型在下游任务中表现的有效方法,但在数据分布变化时仍难以泛化,即使新数据未完全超出训练域。近期以o1和R1为代表的推理导向模型展现出显著优于非推理模型的表现,凸显了推理对泛化与可靠性的重要性。然而,高质量推理轨迹的标注成本高、主观性强且难以扩展。为此,我们利用强化学习(RL)使模型直接从任务结果中学习推理策略。提出一种流程:大模型生成指导工具调用(如函数调用)和最终答案生成的推理步骤。采用基于组相对策略优化(GRPO)的奖励机制,奖励设计围绕工具准确率和答案正确性,使模型可迭代优化推理与行动。实验表明,该方法显著提升了推理质量与工具调用精度,在相对性能上优于仅使用SFT训练的模型1.5%,较基础版vanilla Qwen3-1.7B模型提升40%。结果证明,通过强化学习统一推理与动作学习,是构建更强大、更具泛化能力对话代理的可行路径。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) has emerged as one of the most effective ways to improve the performance of large language models (LLMs) in downstream tasks. However, SFT can have difficulty generalizing when the underlying data distribution changes, even when the new data does not fall completely outside the training domain. Recent reasoning-focused models such as o1 and R1 have demonstrated consistent gains over their non-reasoning counterparts, highlighting the importance of reasoning for improved generalization and reliability. However, collecting high-quality reasoning traces for SFT remains challenging -- annotations are costly, subjective, and difficult to scale. To address this limitation, we leverage Reinforcement Learning (RL) to enable models to learn reasoning strategies directly from task outcomes. We propose a pipeline in which LLMs generate reasoning steps that guide both the invocation of tools (e.g., function calls) and the final answer generation for conversational agents. Our method employs Group Relative Policy Optimization (GRPO) with rewards designed around tool accuracy and answer correctness, allowing the model to iteratively refine its reasoning and actions. Experimental results demonstrate that our approach improves both the quality of reasoning and the precision of tool invocations, achieving a 1.5% relative improvement over the SFT model (trained without explicit thinking) and a 40% gain compared to the base of the vanilla Qwen3-1.7B model. These findings demonstrate the promise of unifying reasoning and action learning through RL to build more capable and generalizable conversational agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。