让工具型智能体更懂行：用细粒度奖励提升金融咨询任务准确率
ToolRLA: Multiplicative Reward Decomposition for Tool-Integrated Agents
- 设计乘法式奖励分解，分四维评估工具调用正确性
- 任务完成率提至91%，工具错误和违规率大幅下降
- 适合高风险领域部署，对复杂工具链优化有实用价值
将推理与API调用结合的工具型智能体在处理复杂任务方面具有潜力，但在高风险、特定领域部署中对齐仍具挑战：现有强化学习方法依赖粗粒度二值奖励，无法区分工具选择错误与参数格式错误。本文提出ToolRLA，一种三阶段后训练流程（SFT -> GRPO -> DPO），核心是细粒度奖励函数，采用乘法形式分解四个维度——格式有效性、工具选择、参数准确性与合规性，并将领域优先级编码为奖励空间中的归纳偏置。在金融顾问协作者系统中部署（80+顾问，每日超1200次查询），ToolRLA实现：任务完成率从62%提升至91%（+47%），工具调用错误率从38%降至14%（-63%），合规违规率从12%降至0.8%（-93%），延迟低于2秒。消融实验表明，乘法奖励设计比加法方案贡献额外7个百分点性能提升。通用性在ToolBench与API-Bank上得到验证。
原文摘要 · Abstract (English)
Tool-integrated agents that interleave reasoning with API calls are promising for complex tasks, yet aligning them for high-stakes, domain-specific deployment remains challenging: existing reinforcement learning approaches rely on coarse binary rewards that cannot distinguish tool selection errors from malformed parameters. We present ToolRLA, a three-stage post-training pipeline (SFT -> GRPO -> DPO) for domain-specific tool agents. The core contribution is a fine-grained reward function with multiplicative correctness decomposition spanning four dimensions -- format validity, tool selection, parameter accuracy, and regulatory compliance -- that encodes domain priority orderings as inductive biases in the reward landscape. Deployed on a financial advisory copilot (80+ advisors, 1,200+ daily queries), ToolRLA achieves over three months: a 47% improvement in task completion rate (62%->91%), a 63% reduction in tool invocation errors (38%->14%), and a 93% reduction in regulatory violations (12%->0.8%), within sub-2-second latency. Ablation studies show the multiplicative reward design accounts for 7 percentage points of improvement over additive alternatives. Generalization is further validated on ToolBench and API-Bank.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。