arXiv:2606.12499cs.RO2026-06被引 2

用视觉动作历史预训练,让机器人更懂操作的前后因果。

Action-Effect Memory Pretraining for Robot Manipulation

论文配图:Action-Effect Memory Pretraining for Robot Manipulation
图 1 · 摘自论文原文
  • 将视觉与动作特征交错编码,通过掩码建模学习动作驱动的状态演变。
  • 在模拟和真实场景中均提升抓取性能,尤其在杂乱和非马尔可夫任务中显著领先。
  • 仅用单向量存储历史信息,高效且适合实时控制应用。

我们提出AEM,一种面向机器人操作的行动-效应记忆预训练框架,从视觉-动作历史中学习紧凑的时序表征。与以往侧重单帧视觉编码的方法不同,AEM关注操作的时序特性——在部分可观测情况下,仅凭当前观测往往不足。AEM将操作建模为动作驱动的交互过程,通过交错视觉与动作特征,并应用掩码建模从不完整的历史中恢复缺失内容,从而学习动作条件下的状态演化。最终视觉标记的Mamba编码输出作为紧凑的历史表征,作为解码与下游控制的全局上下文。该设计保持单向量时序瓶颈的同时确保推理高效。我们在Diffusion Policy和Flow Policy上评估AEM,结果表明其在仿真与真实世界中均持续提升操作性能,在干净、杂乱及随机场景、非马尔可夫任务中均优于基线。消融实验进一步显示,基于历史的预训练优于单帧预训练与直接帧堆叠,同时降低推理延迟与计算开销。

原文摘要 · Abstract (English)

We present AEM, an Action-Effect Memory pretraining framework for robot manipulation that learns compact temporal representations from vision-action history. Unlike prior robot representation pretraining methods that mainly focus on single-frame visual encoding, AEM targets the temporal nature of manipulation, where the current observation alone is often insufficient under partial observability. AEM models manipulation as an action-driven interaction process by interleaving visual and action features and applying masked modeling to recover missing content from incomplete histories, thereby learning action-conditioned state evolution. The Mamba-encoded output of the final vision token is used as a compact history representation, serving as the global context for decoding and downstream control. This design preserves a single-vector temporal bottleneck while keeping inference efficient. We evaluate AEM with Diffusion Policy and Flow Policy. AEM consistently improves manipulation performance in both simulation and real-world settings, outperforming baselines across clean scenes, cluttered and random scenes, and non-Markovian tasks. Ablation studies further show that history-aware pretraining surpasses single-frame pretraining and direct frame stacking, while reducing inference latency and computational cost.

机器人操作时序建模预训练记忆网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。