将语言知识外置到可扩展的记忆模块,提升机器人执行复杂指令的能力。
Key-Gram: Extensible World Knowledge for Embodied Manipulation

- 用哈希查找提取任务关键词,通过门控融合注入视觉主干网络。
- 在多个数据集上实现平均29.5%和9.9%的性能提升,无需微调。
- 适合需要持续更新指令知识的机器人长期任务场景。
具身控制需在动态视觉状态中理解复合语言指令。现有视觉-语言-动作策略常将语言知识与视觉计算耦合于共享主干,导致模态竞争且知识扩展依赖主干更新。本文提出Key-Gram,一种条件记忆框架,将语言驱动的世界知识与视觉状态推理分离。其核心为记忆模块,将指令分解为任务特定的关键词,通过确定性哈希查找获取静态语言先验,并利用上下文感知门控与轻量卷积融合注入选定隐藏层。该设计使主干专注视觉推理与动作推断,可复用的指令知识则存储于可扩展的外部记忆中。逻辑记忆表训练时可分区,因$O(1)$查找特性,推理时可高效置于主机内存。在RoboTwin2.0、LIBERO/LIBERO-Plus及真实双臂操作任务中,Key-Gram持续提升$π_{0}$和$π_{0.5}$主干,分别取得平均29.5%/9.9%(RoboTwin2.0)、35.8%/4.5%(LIBERO-Plus跨域迁移无微调)、15.4%/8.1%(真实世界长程任务)的相对增益。结果表明,外置语言记忆能有效提升复合语义对齐、迁移能力与真实世界操作性能。
原文摘要 · Abstract (English)
Embodied control increasingly requires models to follow compositional language instructions while reasoning over dynamic visual states. However, current vision-language-action policies and world-action models often couple linguistic knowledge with visual computation in a shared backbone or conditioning pathway, leading to modality competition and making knowledge extension dependent on backbone updates. In this paper, we introduce Key-Gram, a conditional-memory framework that separates language-derived world knowledge from visual-state reasoning for embodied control. At its core is a memory module that decomposes an instruction into task-specific key-grams, retrieves static linguistic priors through deterministic hashed lookup, and injects the retrieved entries into selected hidden layers through context-aware gating and lightweight convolutional fusion. This design allows the backbone to devote its main capacity to visual reasoning and action inference, while reusable instruction knowledge is stored in an extensible external memory. The logical memory table can be conveniently partitioned during training and, due to its $O(1)$ lookup pattern, efficiently placed on host memory during inference. Across RoboTwin2.0, LIBERO/LIBERO-Plus, and real-world dual-arm manipulation, Key-Gram consistently improves both $π_{0}$ and $π_{0.5}$ backbones, with average relative gains of $29.5\%/9.9\%$ on RoboTwin2.0, $35.8\%/4.5\%$ on LIBERO-Plus transfer without target-domain fine-tuning, and $15.4\%/8.1\%$ on real-world long-horizon tasks. These results demonstrate that externalized linguistic memory provides an effective and extensible mechanism for improving compositional grounding, transfer, and real-world manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。