让智能体自动学习如何整理经验技能,实现自我进化。
SkillOS: Learning Skill Curation for Self-Evolving Agents

- 用可训练的技能管家搭配冻结的执行器,从经验中持续优化技能库。
- 在多轮任务中表现优于基线,效率与效果均提升,且跨模型和任务泛化好。
- 适合需要长期积累与自主进化的智能体系统,如复杂决策代理。
基于大模型的智能体越来越多用于处理流式任务,但往往只是一次性求解者,无法从过往交互中学习。从经验中提炼出的可复用技能为自我进化提供了自然基础,而高质量的技能整理正是关键瓶颈。现有方法或依赖人工整理、或采用启发式操作、或仅训练短期技能,难以从间接且延迟的反馈中学习复杂的长期整理策略。为此,我们提出 SkillOS,一种面向自演化智能体的、基于经验的强化学习训练方案,用于学习技能整理。SkillOS 将一个固定的执行器(负责检索与应用技能)与一个可训练的技能管家(负责更新外部技能仓库)配对。为提供整理的学习信号,设计了复合奖励,并基于与技能相关的任务依赖关系,在分组的任务流上进行训练:早期轨迹更新技能仓库,后续相关任务评估这些更新。在多轮代理任务与单轮推理任务中,SkillOS 均显著优于无记忆与强记忆基线,在有效性和效率上均有提升,且所学的技能管家可在不同执行器架构与任务领域间泛化。进一步分析表明,学习到的管家能更精准地调用技能,而技能仓库中的技能也逐渐演变为结构更丰富的 Markdown 文件,编码更高阶的元技能。
原文摘要 · Abstract (English)
LLM-based agents are increasingly deployed to handle streaming tasks, yet they often remain one-off problem solvers that fail to learn from past interactions. Reusable skills distilled from experience provide a natural substrate for self-evolution, where high-quality skill curation serves as the key bottleneck. Existing approaches either rely on manual skill curation, prescribe heuristic skill operations, or train for short-horizon skill operations. However, they still struggle to learn complex long-term curation policies from indirect and delayed feedback. To tackle this challenge, we propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents. SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience. To provide learning signals for curation, we design composite rewards and train on grouped task streams based on skill-relevant task dependencies, where earlier trajectories update the SkillRepo, and later related tasks evaluate these updates. Across multi-turn agentic tasks and single-turn reasoning tasks, SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency, with the learned skill curator generalizing across different executor backbones and task domains. Further analyses show that the learned curator produces more targeted skill use, while the skills in SkillRepo evolve into more richly structured Markdown files that encode higher-level meta-skills over time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。