提出动态双粒度技能库，提升智能体强化学习的可复用经验利用效率。
Dynamic Dual-Granularity Skill Bank for Agentic RL
- 将经验分为任务级与步骤级技能，实现高层引导与精细决策支持
- 通过对比基线与技能注入的性能差，生成回溯性奖励信号优化策略
- 技能库可自更新、按效检索与裁剪，适合多场景智能体应用
智能体强化学习可显著受益于可复用的经验，但现有基于技能的方法主要提取轨迹级指导，缺乏对动态演变技能记忆的系统性维护机制。本文提出 D2Skill，一种面向智能体强化学习的动态双粒度技能库，将可复用经验组织为任务技能（用于高层引导）和步骤技能（用于精细决策与错误修正）。D2Skill 通过同一策略下基线与技能注入的对比采样，利用两者性能差距生成回溯性效用信号，联合优化策略与技能库。技能库完全由训练期经验构建，通过反思持续扩展，并采用效用感知的检索与剪枝机制维持高效性。在 ALFWorld、WebShop 及 Search-Augmented QA 任务上的实验表明，D2Skill 在不同规模模型上均显著优于无技能基线。消融分析显示，双粒度建模与动态维护机制对性能提升至关重要，且所学技能具有更高效用、良好跨场景迁移能力，仅引入轻微训练开销。
原文摘要 · Abstract (English)
Agentic RL can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld, WebShop, and Search-Augmented QA tasks show that D2Skill substantially improves performance over skill-free baselines across models of different scales. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。