让机器人像人一样记住任务细节，解决长期复杂操作难题。
MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation
- 模拟人类工作记忆与长期记忆，构建感知-认知记忆库
- 在真实机器人上实现84%成功率，长任务提升26个百分点
- 适合需要长时间记忆和复杂决策的机器人操控场景
时间上下文对机器人操作至关重要，因为这类任务本质上是非马尔可夫的，但主流视觉-语言-动作（VLA）模型通常忽略这一点，在长周期、依赖时间的任务上表现不佳。认知科学表明，人类依靠工作记忆缓冲短期表征以进行即时控制，而海马系统则保存过去经验的详细内容和语义要点以实现长期记忆。受此启发，我们提出MemoryVLA，一种认知-记忆-动作框架，用于长周期机器人操作。预训练的视觉语言模型将观测编码为感知与认知标记，构成工作记忆；同时，感知-认知记忆库存储来自这些标记的低层细节与高层语义。工作记忆从库中检索决策相关条目，自适应融合当前标记并更新库以合并冗余信息。利用这些标记，一个记忆条件扩散动作专家生成具备时间感知的动作序列。我们在三个机器人上的150多个仿真与真实世界任务中评估了MemoryVLA。在SimplerEnv-Bridge、Fractal、LIBERO-5套件和Mikasa-Robo上，其成功率分别为71.9%、72.7%、96.5%和41.2%，均优于最先进基线CogACT和pi-0，其中Bridge任务提升14.6个百分点，Mikasa-Robo提升11.8个百分点。在12个涵盖通用技能与长周期时间依赖的真实任务中，成功率达到84.0%，长周期任务相比基线提升26个百分点。
原文摘要 · Abstract (English)
Temporal context is essential for robotic manipulation because such tasks are inherently non-Markovian, yet mainstream VLA models typically overlook it and struggle with long-horizon, temporally dependent tasks. Cognitive science suggests that humans rely on working memory to buffer short-lived representations for immediate control, while the hippocampal system preserves verbatim episodic details and semantic gist of past experience for long-term memory. Inspired by these mechanisms, we propose MemoryVLA, a Cognition-Memory-Action framework for long-horizon robotic manipulation. A pretrained VLM encodes the observation into perceptual and cognitive tokens that form working memory, while a Perceptual-Cognitive Memory Bank stores low-level details and high-level semantics consolidated from it. Working memory retrieves decision-relevant entries from the bank, adaptively fuses them with current tokens, and updates the bank by merging redundancies. Using these tokens, a memory-conditioned diffusion action expert yields temporally aware action sequences. We evaluate MemoryVLA on 150+ simulation and real-world tasks across three robots. On SimplerEnv-Bridge, Fractal, LIBERO-5 suites and Mikasa-Robo, it achieves 71.9%, 72.7%, 96.5%, and 41.2% success rates, respectively, all outperforming state-of-the-art baselines CogACT and pi-0, with a notable +14.6 gain on Bridge and +11.8 gain on Mikasa-Robo. On 12 real-world tasks spanning general skills and long-horizon temporal dependencies, MemoryVLA achieves 84.0% success rate, with long-horizon tasks showing a +26 improvement over state-of-the-art baseline. Project Page: https://shihao1895.github.io/MemoryVLA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。