让机器人视频策略记住过去，提升长序列操作成功率
MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation
- 引入记忆模块，用压缩帧嵌入和任务进度判断生成历史信息
- 在真实机器人上实现75%以上操作成功率，最高达80%
- 无需逐帧标注，适用于多种模型架构，适合长时序机器人任务
视频世界模型策略通过预测未来观测来学习与动作相关的表征，但仅依赖短期观察窗口，导致长时序操作非马尔可夫性——正确动作依赖于已不可见的早期事件。本文提出MemoryVAM，一种为视频世界模型策略设计的事件记忆机制。采用基于Perceiver的摘要压缩器（Recap-Cue, RC）模块，将每帧CLIP嵌入压缩为紧凑的记忆令牌，并通过轻量级提示门（Cue Gate）从记忆和语言中估计任务完成度。这些令牌被注入视频主干和动作解码器，使策略想象与任务进展对齐，实现基于历史的动作决策。模型通过视频预测、增量重构辅助损失及剧集边界监督训练，无需逐帧进度标签。该机制可适配UNet与扩散变压器（DiT）主干，仅需修改交叉注意力注入接口。在LIBERO-Mem数据集上，平均成功率从5%提升至42.5%；在真实机器人上，计数任务成功率78.3%，空间回忆任务80.0%，顺序追踪任务75.0%。
原文摘要 · Abstract (English)
Video-world-model policies learn action-relevant representations by predicting future observations. However, they condition on only a short observation window, which renders long-horizon manipulation non-Markovian when the correct action depends on earlier events that are no longer visible. We present MemoryVAM, an episodic memory mechanism for video-world-model policies. We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap Compressor maps per-frame CLIP embeddings into compact memory tokens, and a lightweight Cue Gate estimates task completion from memory and language. These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history. Our model trains the memory module with video prediction, a delta-reconstruction auxiliary loss, and episode-boundary supervision, requiring no per-frame progress labels. The same mechanism applies to UNet and Diffusion Transformer (DiT) backbones by changing only the cross-attention injection interface. On LIBERO-Mem, our model improves average success from 5% to 42.5%. On real robots, it achieves 78.3% success on counting tasks, 80.0% on spatial recall, and 75.0% on sequential tracking. Project page: https://MemoryVAM.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。