让大模型智能体自主发现并内化技能,无需外部工具或推理时调用。
SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training

- 通过自挖掘方式从自身成功轨迹中提炼技能,不依赖外部生成器。
- 在ALFWorld和WebShop上分别提升性能至0.930和0.813,优于基线方法。
- 适合追求低延迟、无外部依赖的长周期任务大模型部署场景。
长时序大模型智能体可受益于可复用技能,但现有基于技能的方法通常在训练中依赖外部技能生成器或推理时持续调用技能库,增加工程复杂度、上下文长度与部署延迟。我们提出自内化强化学习框架SIRI,包含三个阶段:首先用GiGPO预热策略,获取基础交互能力并收集无技能成功轨迹;随后进行自技能挖掘,当前策略从自身成功的普通轨迹中总结紧凑技能,并通过配对的带技能与无技能轨迹验证其有效性;最后,仅将有益的技能引导动作标记蒸馏回普通策略,依据轨迹级效用与动作级优势。推理时,智能体仅使用原始提示。在ALFWorld与WebShop上使用Qwen2.5-7B-Instruct测试,SIRI将GiGPO性能从0.908提升至0.930(ALFWorld),从0.728提升至0.813(WebShop),超越提示、强化学习与记忆增强类基线。进一步分析表明,自挖掘策略性能可媲美使用闭源大模型的蒸馏结果。代码已开源。
原文摘要 · Abstract (English)
Long-horizon LLM agents can benefit from reusable skills, yet existing skill-based methods often rely on external skill generators during training or persistent skill retrieval at inference, increasing engineering complexity, context length, and deployment latency. We propose Self-Internalizing Reinforcement learning with Intrinsic skills (SIRI), a three-phase framework that enables agents to discover, validate, and internalize skills without external skill generators or inference-time skill banks. SIRI first warms up the policy with GiGPO to acquire basic interaction ability and collect successful skill-free trajectories. It then performs self-skill mining, where the current policy summarizes compact skills from its own successful plain rollouts and validates them through paired skill-augmented and skill-free rollouts. Finally, SIRI distills only beneficial skill-guided action tokens into the plain policy using trajectory-level utility and action-level advantage. At inference, the agent runs with the original prompt only. On ALFWorld and WebShop with Qwen2.5-7B-Instruct, SIRI improves GiGPO from 0.908 to 0.930 on ALFWorld and from 0.728 to 0.813 on WebShop, outperforming prompt-based, RL-based, and memory-augmented baselines. Further analysis shows that our self-mining strategy can achieve performance comparable to distillation with closed-source large model. Our code is available at https://github.com/kirito618/SIRI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。