通过拓扑结构分析工具使用中的交互动态,提升智能体训练效果。
TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training
- 将多轮实验轨迹映射到统一语义拓扑,捕捉工具操作与环境反馈的演化关系。
- 在SFT和RL中分别优先选择有纠错能力、策略多样性的轨迹与高错误分支任务,性能提升4.2%和6.9%。
- 适合研究具身智能、工具使用与强化学习中稀疏奖励问题的开发者参考。
训练工具使用智能体通常依赖结果导向的筛选:在成功轨迹上进行监督微调(SFT),在通过率高的任务上进行强化学习(RL)。然而,这一范式忽视了交互动态:成功轨迹可能缺乏错误恢复或存在冗余,而通过率无法区分结构信息丰富的任务与简单任务。我们提出 extbf{TopoCurate},一个感知交互的框架,将同一任务下的多轮滚动轨迹投影到统一的语义商拓扑中。通过合并等价的动作-观测状态,该投影将散乱的线性轨迹转化为结构化的流形,明确捕捉工具调用与环境响应如何驱动有效策略与失败模式的分化。基于此表示,我们引入双选机制:对于SFT,优先选择展现反思性恢复、语义高效性和策略多样性的轨迹,缓解协变量偏移与模式崩溃;对于RL,选择具有高错误分支比例和策略异质性的任务,最大化梯度信噪比,解决稀疏奖励场景下的信号消失问题。在BFCLv3和Tau2 Bench上的评估显示,TopoCurate相较于最先进基线分别获得4.2%(SFT)和6.9%(RL)的持续提升。代码与数据即将发布,供进一步研究。
原文摘要 · Abstract (English)
Training tool-use agents typically relies on outcome-based filtering: Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks. However, this paradigm ignores interaction dynamics: successful trajectories may lack error recovery or exhibit redundancy, while pass rates fail to distinguish structurally informative tasks from trivial ones. We propose \textbf{TopoCurate}, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology. By merging equivalent action-observation states, this projection transforms scattered linear trajectories into a structured manifold that explicitly captures how tool invocations and environmental responses drive the divergence between effective strategies and failure modes. Leveraging this representation, we introduce a dual-selection mechanism: for SFT, we prioritize trajectories demonstrating reflective recovery, semantic efficiency, and strategic diversity to mitigate covariate shift and mode collapse; for RL, we select tasks with high error branch ratios and strategic heterogeneity, maximizing gradient Signal-to-Noise Ratio to address vanishing signals in sparse-reward settings. Evaluations on BFCLv3 and Tau2 Bench show that TopoCurate achieves consistent gains of 4.2\% (SFT) and 6.9\% (RL) over state-of-the-art baselines. We will release the code and data soon for further investigations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。