提出在线验证技能有效性的框架,提升语言智能体的技能复用效率。
Co-Evolving Skill Generation and Policy Optimization

- 通过对比基线与加入候选技能的轨迹,评估技能的边际效用。
- 在不增加采样成本的前提下,筛选出有用技能并淘汰有害技能。
- 让策略模型自动生成技能,减少对大模型调用的依赖。
技能增强的强化学习通过存储从过往经验中获取的可复用过程知识,提升了语言智能体的能力。现有方法通常利用强大的语言模型分析轨迹、生成技能,并在在线训练中更新可检索的技能库。然而,它们很少在存储前评估新生成技能的实际效用。我们发现这一假设不可靠:即使由前沿大模型生成的技能,其效用也参差不齐,许多技能几乎无益甚至损害性能。一旦这些技能进入技能库,其影响难以追溯,因为后续回放反馈延迟且反映的是多个技能的综合效果,而非单个技能的边际贡献。为此,我们提出一种在线强化学习框架,用于技能存储前的验证。该框架估计候选技能是否在当前任务和检索上下文中提供了额外信息。它利用标准回放预算,在相同任务与检索背景下构建两组匹配轨迹:仅基于已检索技能的基线回放,以及在相同技能基础上再加入候选技能的增强回放。两组间的奖励差距即为候选技能的上下文相关边际效用,从而实现对有效技能的促进与无效或有害技能的过滤,且无需额外回放开销。框架进一步利用此边际效用信号训练策略本身作为技能生成器,降低对频繁调用专有模型的依赖。所学的技能生成概率作为上下文相关的评分,可用于检索时的重排序与过时技能的修剪,以适应策略的演化。
原文摘要 · Abstract (English)
Skill-augmented reinforcement learning improves language agents by storing reusable procedural knowledge acquired from past experience. Existing methods typically use strong language models to analyze trajectories, generate skills, and update a retrievable skill bank during online training. However, they rarely assess whether a newly generated skill is useful before it is stored and reused. We find that this assumption is unreliable: even skills generated by proprietary frontier LLMs exhibit highly mixed utility, with many providing little benefit or even degrading performance. Once such skills enter the bank, their effects are difficult to identify, because subsequent rollout feedback is delayed and usually reflects the combined effect of multiple retrieved skills rather than the marginal contribution of any individual skill. We propose an online reinforcement learning framework for pre-storage skill validation. The framework estimates whether a candidate skill contributes useful information beyond the skills already retrieved for the current task. It uses the standard rollout budget to form two matched groups under the same task and retrieval context: base rollouts conditioned on the currently retrieved skills, and skill-augmented rollouts conditioned on the same skills plus one candidate skill induced from the base trajectories. The reward gap between these two groups estimates the candidate skill's context-dependent marginal utility, enabling the framework to promote useful skills while filtering ineffective or harmful ones without additional rollout overhead. The framework further uses this marginal-utility signal to train the policy itself as a skill generator, reducing reliance on repeated calls to proprietary models. The learned skill-generation likelihood serves as a context-dependent score for retrieval-time reranking and outdated-skill pruning as the policy evolves.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。