提升大模型工具使用效率,精准区分何时该用工具
Efficient Agentic Reinforcement Learning with On-Policy Intrinsic Knowledge Boundary Enhancement

- 训练时动态双路径对比,识别模型是否需工具
- 平均准确率提升1.85点,工具调用减少18%
- 可适配多种强化学习算法,适合优化智能体决策
基于大模型的智能体强化学习在使用外部工具时表现良好,但存在冗余调用和内在知识边界模糊的问题,即模型难以判断何时应使用工具、何时仅靠参数化知识即可。现有基于奖励塑造的方法优化目标粗粒度,易导致盲目抑制工具调用,引发奖励劫持。本文提出AKBE(Agentic Knowledge Boundary Enhancement),一种基于策略的动态方法,在训练中通过带工具与不带工具的双路径回放探测模型内在知识边界。将知识边界定义为每个实例是否需要工具及最少调用次数。通过比较不同路径的正确性,对轨迹进行分类并生成针对性监督信号,融入强化学习训练流程。在七个问答基准测试中,AKBE平均任务准确率提升1.85,工具调用减少18%,工具利用率提高25%,无准确率-效率权衡。进一步分析表明其可兼容多种强化学习算法,且各类信号机制清晰。代码已开源。
原文摘要 · Abstract (English)
Agentic reinforcement learning (RL) has proven effective for training LLM-based agents with external tool-use capabilities. However, we identify that agentic RL training induces increasing redundant tool calls and blurs the model's intrinsic knowledge boundary, where the model fails to distinguish when tools are needed versus when parametric knowledge suffices. Existing solutions based on reward shaping create coarse-grained optimization targets that tend to incentivize indiscriminate tool-call suppression, leading to reward hacking. In this paper, we propose AKBE (Agentic Knowledge Boundary Enhancement), an on-policy method that dynamically probes the model's intrinsic knowledge boundary through dual-path (with-tool and no-tool) rollouts during training. We define the knowledge boundary as the per-instance determination of whether tools are required and the minimum tool calls necessary. By comparing correctness across paths, AKBE categorizes trajectories and constructs targeted supervisory signals that guide efficient tool-use patterns for each question. These signals are integrated seamlessly into the agentic RL training loop. Experiments on seven QA benchmarks demonstrate that AKBE improves task accuracy by +1.85 on average and reduces tool calls by 18% over standard agentic RL, yielding 25% higher tool productivity without any accuracy-efficiency trade-off. Further analysis suggests its plug-and-play compatibility across different RL algorithms and the mechanism of each signal category. Our code is available at https://github.com/CuSO4-Chen/AKBE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。