arXiv:2508.08791cs.CLcs.AI2025-08ACL被引 16

用自动化环境提升大模型工具使用能力,反馈更精准,训练更高效。

Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments

  • 构建自动化的训练环境,支持场景分解与功能集成
  • 设计可验证奖励机制,提升工具使用准确率与任务完成度
  • 适合研究大模型推理与强化学习的开发者

大语言模型有效使用工具对环境交互至关重要。然而,由于缺乏专为工具使用设计的强化学习框架,进展受限于稳定训练环境构建困难和可验证奖励机制缺失。为此,我们提出自动化环境构建流程,包含场景分解、文档生成、函数集成、复杂度扩展和本地化部署,实现高质量训练环境构建,提供详细可量化的反馈且无需依赖外部工具。同时引入可验证奖励机制,评估工具使用精度与任务执行完整性。结合所收集轨迹数据,该机制可无缝接入标准强化学习算法,实现反馈驱动的模型训练。在不同规模的大模型上实验表明,该方法显著提升工具使用性能,且不损害通用能力。分析显示,性能提升源于上下文理解与推理能力增强,由模型低层MLP参数更新驱动。代码与数据见https://github.com/bytedance/FTRL。

原文摘要 · Abstract (English)

Effective tool use is essential for large language models (LLMs) to interact with their environment. However, progress is limited by the lack of efficient reinforcement learning (RL) frameworks specifically designed for tool use, due to challenges in constructing stable training environments and designing verifiable reward mechanisms. To address this, we propose an automated environment construction pipeline, incorporating scenario decomposition, document generation, function integration, complexity scaling, and localized deployment. This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools. Additionally, we introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution. When combined with trajectory data collected from the constructed environments, this mechanism integrates seamlessly with standard RL algorithms to facilitate feedback-driven model training. Experiments on LLMs of varying scales demonstrate that our approach significantly enhances the models' tool-use performance without degrading their general capabilities. Our analysis suggests that these gains result from improved context understanding and reasoning, driven by updates to the lower-layer MLP parameters in models. Code and data are available at https://github.com/bytedance/FTRL.

大模型强化学习工具使用自动化训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。