让大模型智能体学会复用经验技能,不改参数也能越用越强。
Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

- 通过非参数化PPO从交互中自动提取可重用的技能模块。
- 在压缩内存条件下仍保持高复用率和性能提升。
- 适合需要长期自主决策的AI代理系统研究者。
基于大语言模型的智能体在序列决策中表现优异,但常依赖即时推理,重复推导相同场景解决方案,导致计算冗余与行为不稳定。为此,我们提出Skill-Pro框架,使智能体能无参数更新地从交互经验中自主学习可复用的程序性技能。通过构建技能-马尔可夫决策过程(Skill-MDP),Skill-Pro将被动的回合叙事转化为具备激活、执行与终止条件的可执行技能,确保其可用性。为实现可靠复用且不退化能力,引入非参数化PPO:利用语义梯度生成高质量候选技能,并通过PPO门进行鲁棒验证。基于评分维护机制,技能库保持紧凑且高质量。在同域、跨任务及跨智能体场景下的实验表明,Skill-Pro实现了更高的技能复用率,并在极端内存压缩下仍取得显著性能增益。可视化演进轨迹与技能分布揭示了其透明积累、优化与复用程序知识的过程,推动长期自主性发展。
原文摘要 · Abstract (English)
LLM-driven agents excel at sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and instability. To bridge this gap, we propose Skill-Pro, a framework enabling agents to autonomously learn reusable procedural skills from interaction experiences without parameter updates. By formalizing a Skill-MDP, Skill-Pro transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, Skill-Pro sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that Skill-Pro achieves superior reuse rates and significant gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how Skill-Pro transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。