评测机器人操作中的过程级记忆能力,揭示现有模型记忆缺陷。
MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

- 设计30个依赖历史的任务,通过检查点标注存储、更新、压缩三种记忆操作。
- 标准模型平均成功率仅31.9%,高存储率常伴随弱更新与压缩能力。
- 提供语义对齐监督,助力训练更鲁棒的具身记忆系统,适合机器人研究者。
机器人操作常需处理已不可见的信息,但当前视觉-语言-动作策略多基于即时观测评估,忽视记忆作用。现有基准主要依赖任务成功,混淆遗忘与操作失败。本文提出MEMOBench,一个面向机器人操作过程级记忆的评测基准。包含30个依赖历史的任务、1,500条专家演示及4,200个可执行检查点实例,源自84个模板。每个检查点配以粗粒度到细粒度的语言描述,结合模拟器谓词,标注记忆操作:存储、更新或压缩。这些标注定义了记忆存储率、更新率与压缩率,衡量记忆保真度。在标准与记忆增强型VLA策略中,最强基线平均成功率仅为31.9%,且高存储常伴弱更新与压缩。检查点语言还用于监督语义、对比与帧间记忆对齐目标,带来各记忆操作的适度提升。MEMOBench提供诊断评估工具与训练监督信号,支持具身记忆策略发展。项目页见https://github.com/Collab-Gen/MEMOBench。
原文摘要 · Abstract (English)
Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the current observation largely determines the next action. Existing robotic memory benchmarks expose this gap, but they still rely mainly on final task success and therefore conflate forgetting with manipulation failure. We present \textbf{MEMOBench}, a benchmark for process level memory evaluation in robotic manipulation. MEMOBench includes 30 history dependent tasks, 1{,}500 expert demonstrations, and 4{,}200 executable checkpoint instances from 84 templates. Each checkpoint pairs coarse to fine language with a simulator predicate and labels one memory operation: Storage, Update, or Compression. These annotations define Memory Storage Rate, Memory Update Rate, and Memory Compression Rate, which measure memory fidelity alongside task success. Across standard and memory augmented VLA policies, the strongest memory module baseline reaches only 31.9\% average success rate, and high storage often coexists with weak update and compression. Checkpoint language also supervises semantic, contrastive, and framewise memory alignment objectives, yielding modest gains across different memory operations. MEMOBench provides a diagnostic evaluation suite and training supervision for memory grounded robotic policies. The project page is available at https://github.com/Collab-Gen/MEMOBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。