不用记忆模块,直接传完整视频历史,让视觉语言动作模型高效长时决策。
SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

- 保留原始采样视频序列,以时间戳格式输入预训练骨干网络。
- 在四类记忆基准上达到新SOTA,通用控制任务性能无损失。
- 适合需要长时依赖推理的机器人操作场景,实现低延迟高精度。
长时程操作任务具有部分可观测性:做出下一步动作所需信息可能仅出现在数分钟前的观测中。现有记忆机制如检索库、学习压缩器、循环状态需在未知未来需求前提前决定保留什么历史,这源于认为分钟级历史过大无法直接处理的假设——而现代视觉语言模型骨干网络已不再受限。本文提出SimpleMemVLA,一种无需专用记忆模块的视觉-语言-动作模型。它保持采样历史完整,并以骨干网络预训练所支持的时间戳视频格式传递;生成子任务的隐藏状态作为历史到标准流匹配动作头的唯一通道。由于连续决策共享大部分历史,执行时预先填充共享前缀,使延迟接近单帧VLA。固定骨干与训练设置下,SimpleMemVLA在四个记忆基准上显著优于检索、压缩和循环状态机制,因果干预验证策略确实读取了历史。代码开源于https://github.com/wadeKeith/SimpleMemVLA。
原文摘要 · Abstract (English)
Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earlier. Existing memory mechanisms: retrieval banks, learned compressors, recurrent states must decide what to keep from the past before knowing what a future decision will require. This was motivated by the assumption that minute-scale history is too large to process directly, which modern VLM backbones no longer make true. In this work, we introduce SimpleMemVLA, a VLA without a dedicated memory module. It keeps the sampled history intact and passes it to the backbone in the timestamped video format the backbone was pretrained to process; the hidden states of a generated sub-task then form the only channel from history to a standard flow-matching action head. Since consecutive decisions share most of their history, prefilling the shared prefix during action execution keeps latency close to a single-frame VLA. SimpleMemVLA sets a new state of the art on four memory benchmarks without cost on general-purpose control. Holding the backbone and training setup fixed, it outperforms retrieval, compression and recurrent-state mechanisms by a wide margin, and causal interventions confirm that the policy genuinely reads its history. Code available at https://github.com/wadeKeith/SimpleMemVLA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。