发现大模型推理与工具使用会相互干扰,提出解耦训练方法显著提升性能。
Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning
- 用量化方法揭示推理与工具使用存在梯度冲突
- 解耦训练后在13个任务上超越所有联合优化基线
- 适合追求高精度智能体的开发者和研究者
智能体强化学习(ARL)训练大语言模型将推理与外部工具调用结合以解决复杂任务。现有方法通常共享参数同时优化推理与工具使用,隐含假设联合训练能提升整体性能,但这一假设缺乏实证检验。本文引入能力效应归因(CEA),定量证明推理与工具使用常导致梯度方向错位,造成训练干扰,削弱联合优化效果,挑战当前主流范式。为此,提出解耦动作-推理微调(DART),通过独立的低秩适配模块分离推理与工具使用的参数更新。仅此简单改进,DART在检索增强问答与自然语言转SQL共十三项基准上超越所有联合优化基线,逼近双智能体上限,验证了能力干扰的存在。
原文摘要 · Abstract (English)
Agentic Reinforcement Learning (ARL) trains large language models to interleave reasoning with external tool execution to solve complex tasks. Most existing ARL methods train a single set of parameters to support both reasoning and tool-use behaviors, implicitly assuming that joint training leads to improved overall agent performance. Despite its widespread adoption, this assumption has rarely been examined empirically. In this paper, we systematically examine this assumption by introducing Capability Effect Attribution (CEA), which provides quantitative evidence of interference between reasoning and tool-use behaviors. Through an in-depth analysis, we show that these two capabilities often induce misaligned gradient directions, leading to training interference that undermines the effectiveness of joint optimization and challenges the prevailing ARL paradigm. To address this issue, we propose Disentangled Action--Reasoning Tuning (DART), a simple and efficient framework that explicitly decouples parameter updates for reasoning and tool use via separate low-rank adaptation modules. With this simple change alone, DART outperforms all joint-optimization baselines and approaches the 2-Agent upper bound across thirteen benchmarks on retrieval-augmented QA and NL2SQL, further supporting our finding of capability interference under shared optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。