用三类经验增强智能体训练,提升工具使用能力并减少数据消耗。
E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning
- 融合专家引导、自探索与分支路径,动态优化训练过程。
- 工具任务性能提升6%,合成数据用量不足10%。
- 适合追求高效低耗训练的AI系统开发者。
尽管大语言模型在工具集成推理(TIR)中展现出巨大潜力,现有训练范式仍面临显著局限:零样本强化学习因缺乏先验引导导致探索效率低和模式退化;而基于监督微调再强化学习的方法则受限于高数据成本及由低熵坍塌引发的能力瓶颈。为此,我们提出E3-TIR(增强经验利用),一种面向智能体训练早期阶段的预热范式。具体而言,将训练建模为三种经验类型的动态整合:专家前缀、专家引导与自我探索。通过在专家‘锚点’周围执行多样化分支探索,并采用混合策略优化机制,有效缓解分布偏移问题,解决共享前缀引发的优化冲突。该方法动态扩展模型知识边界,在探索多样性与训练效率间取得良好平衡。实验表明,E3-TIR在工具使用任务上相较传统范式实现6%的性能提升,且合成数据用量低于10%。此外,在综合考量性能、数据成本与训练效率的收益指标(ROI)上,较基线提升1.46倍。代码已开源:https://github.com/yuki-younai/E3-TIR。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) have demonstrated significant potential in Tool-Integrated Reasoning (TIR), existing training paradigms face significant limitations: Zero-RL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3-TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert "anchors" and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model's knowledge boundaries, effectively balancing exploration diversity with training efficiency.Experimental results demonstrate that E3-TIR achieves a 6 performance improvement over traditional paradigms on tool-use tasks, while requiring less than 10 of the synthetic data. Furthermore, in terms of ROI, a comprehensive metric integrating performance, data cost, and training efficiency we achieve a 1.46x gain compared to baselines. Code is available at https://github.com/yuki-younai/E3-TIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。