让视觉语言动作模型实时存储长期视觉记忆,效率更高。
NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation

- 用视觉编码器压缩多视角历史帧为单个记忆标记
- 仿真中成功率从32.4%提升至84.0%,真实机器人达98.7%
- 仅需20%数据量即可达到先进水平,适合数据受限场景
预训练的视觉-语言-动作(VLA)模型如何在高频更新下保持长时程视觉历史,同时不牺牲效率?现有方法依赖外部内存管理,限制了记忆长度或反应速度。为此,我们提出NativeMEM,一种具备长期且实时更新记忆的VLA策略。核心是高效的内存编码方案——原生内存压缩(Native Memory Compression),该方法复用VLA自身的视觉编码器,将每个摄像头视角的历史帧压缩为单一标记。这些记忆标记被添加到输入序列中,使预训练VLA能以极低延迟开销关注长期历史,无需外部规划器或重新初始化内存模块。为对齐记忆标记与预训练策略,我们首先在高需求数据上利用冻结的VLA训练通用记忆分词器,再对VLA进行任务特定微调。NativeMEM持续优于以往方法,在仿真中成功率由32.4%提升至84.0%,真实机器人上最高达98.7%,同时保持低推理延迟和GPU内存占用。值得注意的是,NativeMEM具有高数据效率,仅使用20%训练数据即达成与先前方法相当的效果。
原文摘要 · Abstract (English)
How can pretrained Vision-Language-Action (VLA) models retain long-horizon visual histories with high-frequency updates without sacrificing efficiency? Existing approaches rely on external memory management, which restrains either the memory horizon or the reactiveness of pretrained policies. To this end, we present NativeMEM, a VLA policy that features long-term and real-time updated memory. At its core is an efficient memory encoding scheme, Native Memory Compression, which repurposes the VLA's own vision encoder to compress each historical frame from each camera view into a single token. Appended to the input sequence, these memory tokens enable the pretrained VLA to attend over long-term history with negligible latency overhead, requiring neither an external planner nor a freshly initialized memory module. To align the memory tokens with the pretrained policy, we first develop a generic memory tokenizer under the supervision of a frozen VLA on memory-demanding data, and then unfreeze the VLA for task-specific fine-tuning. NativeMEM consistently outperforms prior methods, boosting success rates from 32.4% to 84.0% in simulation and up to 98.7% on real robots, while maintaining low inference latency and GPU memory usage. Notably, NativeMEM exhibits high data efficiency by achieving competitive results with prior arts using only 20% of the training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。