用推理树生成多步监督信号,让智能体更高效地进化技能。
Branch2Skill: Efficient Skill Evolution Through Reasoning Trees

- 构建推理树,通过对比相似路径提取每步的改进证据。
- 相比旧方法减少73.2%的令牌消耗,性能还更好。
- 适合需要低成本迭代训练的智能体系统研究者。
技能演化通过反馈持续提升智能体能力,失败轨迹常能揭示行为缺陷。但现有方法多依赖单一轨迹,早期推理错误会逐级传播,削弱后续反馈效果,导致需反复滚动、诊断、更新,代价高昂。为此,我们提出Branch2Skill,将单个推理树转化为密集监督信号。针对每个任务,在固定预算下执行蒙特卡洛树搜索,获取多样化推理路径;再将最优路径与共享前缀的兄弟路径对比,提取每一步应保留、修正或规避的推理模式;最后将多步证据提炼为可复用的更新指令,使一次推理树即可提供跨多个步骤的监督,显著减少重复滚动-更新循环。在六个涵盖推理与代理任务的基准上,Branch2Skill持续提升任务表现并增强演化效率。例如,以GPT 5.5为目标模型时,其令牌消耗比SkillOpt低73.2%,且性能更优。结果表明,推理树不仅能实现更有效的轨迹搜索,还可提供更丰富的监督信号以支持更高效的技能提升。代码将公开。
原文摘要 · Abstract (English)
Skill evolution improves agent skills through feedback over time, with failed trajectories often providing informative signals by revealing incomplete or misleading behaviors. However, existing methods mainly rely on single trajectories, where early reasoning errors can propagate through subsequent steps and weaken the feedback available for skill refinement. Consequently, improving skills requires repeated cycles of rollout, diagnosis, and update, incurring substantial token costs. To address this challenge, we introduce Branch2Skill, an efficient framework that transforms a single reasoning tree into dense supervision for skill evolution. For each task or problem, Branch2Skill performs Monte Carlo tree search under a fixed budget to obtain diverse reasoning trajectories, then compares an elite path with sibling alternatives sharing the same prefixes to extract step-wise evidence about which reasoning patterns to retain, revise, or avoid. Finally, Branch2Skill distills multi-step evidence into reusable updates, allowing one reasoning tree to provide supervision across multiple reasoning steps and reducing the need for repeated rollout-update cycles. Across six benchmarks covering reasoning and agentic tasks, Branch2Skill consistently improves task performance while enhancing skill evolution efficiency. For example, with GPT 5.5 as the target model, Branch2Skill uses 73.2% fewer tokens than SkillOpt, while achieving superior performance. These results demonstrate that reasoning trees can support not only more effective trajectory search, but also richer supervision for more efficient skill improvement. Code will be published.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。