通过偏差点偏好学习，让工具调用智能体自进化提升多轮任务表现。
Self-Evolution for Multi-Turn Tool-Calling Agents via Divergence-Point Preference Learning
- 基于状态匹配与前缀对齐定位分歧点，构建偏好数据对。
- 在tau2-bench上平均奖励从0.304提升至0.355，增幅16.8%。
- 特别适用于航空、零售等复杂任务场景，适合研究多轮推理与工具调度。
多轮工具调用智能体需协调长序列工具操作，同时追踪对话状态与策略约束。现有方法常将推理时编排与参数级学习分离，导致工具选择结构松散，偏好更新易受训练-部署提示不一致影响。针对基准内自改进，ToolGraph结合模式导出的拓扑结构、成功回放估算的转移权重，以及历史感知的写入前置条件与重复搜索循环控制。我们通过状态匹配与前缀对齐定位分歧点，构建161组偏好对，并经由动作正确性标注过滤，再在与推理相同的ToolGraph上下文中训练DPO。在375个tau2-bench任务中，ToolGraph将加权平均奖励从0.304提升至0.338（相对+11.2%），ToolGraph+DPO达0.355（相对于基线+16.8%），其中DPO增益主要集中在航空与零售任务。细粒度诊断显示，约一半电信轨迹在执行动作前耗尽步数预算，且所选奖励正值是16种评估的DPO配置中最有效的检查点信号。
原文摘要 · Abstract (English)
Multi-turn tool-using agents must coordinate long-horizon tool sequences while tracking dialogue state and policy constraints. Existing approaches often separate inference-time orchestration from parameter-level learning, leaving tool selection weakly structured and preference updates vulnerable to train--deployment prompt mismatch. For within-benchmark self-improvement, ToolGraph combines schema-derived topology, transition weights estimated from successful rollouts, and history-aware controls for write prerequisites and repeated-search loops. We then construct 161 preference pairs by locating divergence points via state-based matching and prefix-based alignment, filtered through action-correctness annotations, and train DPO under the same ToolGraph context used at inference. Across 375 tau2-bench tasks, ToolGraph raises the weighted average reward from 0.304 to 0.338 (+11.2% relative), while ToolGraph+DPO reaches 0.355 (+16.8% over the baseline), with the DPO gain concentrated in airline and retail. Fine-grained diagnostics further show that roughly half of telecom trajectories exhaust the step budget before action execution and that chosen reward positivity is the most useful checkpoint signal across our 16 evaluated DPO configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。