通过专家引导的技能记忆框架,让机器人学会拆解复杂动作并复用基础技能。
SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

- 用专家指导的混合专家模型自动分解长序列动作成原子技能。
- 构建动态记忆库存储技能特征,在推理时检索最相关技能提升预测准确率。
- 显著提升机器人在未见过任务中的组合泛化能力,适合复杂操作场景。
具身视觉运动模型(如扩散策略和视觉-语言-动作模型)在机器人操作基准上表现良好,但受限于大规模具身轨迹数据集稀缺,导致在分布外场景下组合泛化能力不足,难以捕捉可复用的技能结构。为此,我们提出技能记忆框架SkillMemo,将长时程示范隐式分解为潜在原子技能,并将技能级特征融入动态情景记忆库以解决组合任务。具体地,设计基于混合专家(MoE)架构的专家引导轨迹分割模块,通过学习到的门控系数隐式划分轨迹为不同技能单元;进一步构建技能级情景记忆架构,以可检索的键值对形式存储紧凑的技能表示。推理时,记忆库检索最相关的技能原型,并与模型当前的门控分布融合,提供鲁棒上下文先验以优化动作预测。在仿真基准和真实世界操作任务上的大量实验表明,SkillMemo持续提升扩散策略和视觉-语言-动作模型性能,达到当前最优水平,超越$π_{0.5}$基线,展现出对未见任务配置的强大组合泛化能力。
原文摘要 · Abstract (English)
Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures. To address this limitation, we propose Skill-Based Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic memory bank for solving compositional tasks. Specifically, we first introduce an expert-guided trajectory segmentation module built upon a Mixture-of-Experts (MoE) architecture, which implicitly partitions trajectories into distinct skill primitives represented by learned gating coefficients. We further design a skill-level episodic memory architecture that stores compact skill representations as retrievable key-value pairs. During inference, the memory bank retrieves the most relevant skill primitives which are subsequently fused with the model's current gating distribution, providing a robust contextual prior to refine action predictions. Extensive experiments on the simulation benchmark and real-world manipulation tasks demonstrate that SkillMemo consistently enhances both DP and VLA backbones, achieving state-of-the-art performance and outperforming $π_{0.5}$, while exhibiting strong compositional generalization to unseen task configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。