通过奖励标记控制生成质量，提升多轮工具调用的强化学习效果。
RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents
- 用奖励标记引导模型生成不同质量轨迹，实现可控探索。
- 在BFCLv4上优于基线，Qwen-2.5-7B性能超越闭源API模型。
- 适合需要稳定多轮工具调用的智能体研发者使用。
多轮工具调用对大语言模型极具挑战，因奖励稀疏且探索成本高。常用SFT+GRPO方法在组内奖励差异小时会停滞（如多条轨迹获全0或全1奖励），导致组归一化优势信息不足，更新消失。为此，我们提出RC-GRPO（奖励条件分组相对策略优化），将探索视为可控制的导向问题，引入离散奖励标记。首先，在混合质量轨迹上微调奖励条件轨迹策略（RCTP），在提示中注入奖励目标标记（如<|high_reward|>、<|low_reward|>），使模型学会按需生成不同质量轨迹。随后在强化学习中，每组内采样多样奖励标记并条件化轨迹生成，增强组内多样性，提升优势增益。在伯克利函数调用排行榜v4（BFCLv4）多轮基准测试中，该方法持续优于基线，Qwen-2.5-7B-Instruct性能甚至超过所有闭源API模型。
原文摘要 · Abstract (English)
Multi-turn tool calling is challenging for Large Language Models (LLMs) because rewards are sparse and exploration is expensive. A common recipe, SFT followed by GRPO, can stall when within-group reward variation is low (e.g., more rollouts in a group receive the all 0 or all 1 reward), making the group-normalized advantage uninformative and yielding vanishing updates. To address this problem, we propose RC-GRPO (Reward-Conditioned Group Relative Policy Optimization), which treats exploration as a controllable steering problem via discrete reward tokens. We first fine-tune a Reward-Conditioned Trajectory Policy (RCTP) on mixed-quality trajectories with reward goal special tokens (e.g., <|high_reward|>, <|low_reward|>) injected into the prompts, enabling the model to learn how to generate distinct quality trajectories on demand. Then during RL, we sample diverse reward tokens within each GRPO group and condition rollouts on the sampled token to improve within-group diversity, improving advantage gains. On the Berkeley Function Calling Leaderboard v4 (BFCLv4) multi-turn benchmark, our method yields consistently improved performance than baselines, and the performance on Qwen-2.5-7B-Instruct even surpasses all closed-source API models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。