让智能体学会何时不用工具，避免滥用同时保留有用探索。
Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning
- 引入无工具轨迹，按题目难度动态惩罚多余工具调用
- 在多个推理基准上提升准确率效率，工具调用减少超18%
- 适合需要高效使用外部工具的自主智能体研究者
自主强化学习可能导致工具滥用，即模型在仅靠内部推理即可解决的问题上仍过度调用外部工具。现有方法通过统一的工具使用惩罚或硬性限制来缓解此问题，但可能抑制有用的工具辅助探索。我们提出EAPO——一种高效的自主策略优化框架，通过在每轮采样中引入无工具轨迹，采用难度感知奖励重塑，在较简单问题上主要惩罚冗余工具调用，并结合置信度感知的词元重加权以改善策略学习。在九个数学与知识密集型推理基准上，EAPO在Qwen2.5-3B、Qwen2.5-7B和Llama3.1-8B上均显著提升准确率与效率的平衡。相比GRPO，平均性能分别提升10.45%、7.27%和9.69%，同时平均工具调用次数减少18.33%、18.33%和24.59%。结果表明，智能体可在不损害工具融合推理能力的前提下，学会何时不应使用工具。
原文摘要 · Abstract (English)
Agentic reinforcement learning can induce tool abuse, where models overuse external tools even for queries solvable by internal reasoning. Existing approaches mitigate this issue with uniform tool-use penalties or hard limits, which reduce tool frequency but may also suppress useful tool-assisted exploration. We propose EAPO, an Efficient Agentic Policy Optimization framework that learns selective tool use. EAPO introduces tool-free trajectories into each rollout group, applies difficulty-aware reward shaping to penalize redundant tool calls mainly on easier queries, and uses confidence-aware token reweighting to improve policy learning. Across nine mathematical and knowledge-intensive reasoning benchmarks, EAPO consistently improves the accuracy efficiency trade-off on Qwen2.5-3B, Qwen2.5-7B, and Llama3.1-8B. Compared with GRPO, EAPO improves average performance by 10.45%, 7.27%, and 9.69%, while reducing average tool calls by 18.33%, 18.33%, and 24.59%, respectively. These results show that agents can learn when not to use tools without compromising tool-integrated reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。