让小模型也能稳定用工具，通过动态替换难样本和自生成示范提升表现
ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMs
- 用动态替换难样本+指数学习率衰减，解决小模型训练不稳问题
- 自生成示范机制让小模型能自主学习，仅需0.01额外奖励即可提升效果
- 适合资源有限的轻量级模型应用，尤其提升弱小模型的工具使用能力
使用组相对策略优化（GRPO）训练大语言模型时，小规模模型常因生成不准确而表现不佳，不仅削弱性能提升，还频繁引发中段训练崩溃，影响稳定性和最终效果。为此，我们提出ToolExpander框架，通过两项创新推进资源受限模型的工具导向强化学习：(1) 动态多轮硬采样，动态替换连续10次无正确输出的困难样本为高质量少样本示范，并结合指数学习率衰减策略缓解震荡；(2) 自示范思维，一种改进的GRPO框架，移除KL散度项并调整截断系数，使模型能通过极小额外奖励（0.01）自主生成与分析少样本示例。实验表明，ToolExpander显著提升了各类模型的工具使用能力，尤其在小型弱模型上增强了训练稳定性与整体性能。
原文摘要 · Abstract (English)
Training Large Language Models (LLMs) with Group Relative Policy Optimization (GRPO) encounters a significant challenge: models often fail to produce accurate responses, particularly in small-scale architectures. This limitation not only diminishes performance improvements and undermines the potential of GRPO but also frequently leads to mid-training collapse, adversely affecting stability and final efficacy. To address these issues, we propose ToolExpander, a novel framework that advances tool-oriented reinforcement learning for resource-constrained LLMs through two key innovations:(1) Dynamic Multi-Round Hard Sampling, which dynamically substitutes challenging samples(those without correct outputs over 10 rollouts) with high-quality few-shot demonstrations during training, coupled with an exponential learning rate decay strategy to mitigate oscillations;(2) Self-Exemplifying Thinking, an enhanced GRPO framework that eliminates KL divergence and incorporates adjusted clipping coefficients, encouraging models to autonomously generate and analyze few-shot examples via a minimal additional reward (0.01).Experimental results demonstrate that ToolExpander significantly enhances tool-using capabilities in LLMs, especially in weaker small-scale models, improving both training stability and overall performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。