让视觉语言动作模型学会用历史信息做决策,提升机器人长期任务成功率。
HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy
- 引入时刻标记编码每步感知信息,通过时间对比学习增强时序区分能力。
- 轻量记忆模块融合历史时刻标记,使模型在长序列任务中成功率达76.4%。
- 无需重训练,即可让SOTA模型具备历史感知能力,适合真实世界机器人应用。
机器人操作任务本质上具有时序依赖性:利用历史上下文可显著提升表现。然而,现有视觉-语言-动作模型(VLAs)大多仅依赖当前观测,忽略先前信息。本文提出HAMLET,一种可扩展的框架,将VLAs改造为能关注历史上下文的动作策略。具体地,引入时刻标记以紧凑编码每一步的感知信息,并通过时间对比学习初始化,使其更好捕捉时序特征。随后,采用轻量级记忆模块,将历史时刻标记融合为记忆特征,用于动作预测。实验表明,HAMLET成功将SOTA VLA转化为历史感知策略,在长时序任务中表现优异。在GR00T N1.5基础上,于真实世界历史依赖任务上平均成功率达76.4%,较基线提升47.2%。在RoboCasa Kitchen(100-demo设置)上性能从64.1%提升至66.4%,在LIBERO上从95.6%提升至97.7%,验证其在通用机器人操作基准上的有效性。
原文摘要 · Abstract (English)
Inherently, robotic manipulation tasks are history-dependent: leveraging past context could be beneficial. However, most existing Vision-Language-Action models (VLAs) have been designed without considering this aspect, i.e., they rely solely on the current observation, ignoring preceding context. In this paper, we propose HAMLET, a scalable framework to adapt VLAs to attend to the historical context during action prediction. Specifically, we introduce moment tokens that compactly encode perceptual information at each timestep. Their representations are initialized with time-contrastive learning, allowing them to better capture temporally distinctive aspects. Next, we employ a lightweight memory module that integrates the moment tokens across past timesteps into memory features, which are then leveraged for action prediction. Through empirical evaluation, we show that HAMLET successfully transforms a state-of-the-art VLA into a history-aware policy, especially demonstrating significant improvements on long-horizon tasks that require historical context. In particular, on top of GR00T N1.5, HAMLET achieves an average success rate of 76.4% on history-dependent real-world tasks, surpassing the baseline performance by 47.2%. Furthermore, HAMLET pushes prior art performance from 64.1% to 66.4% on RoboCasa Kitchen (100-demo setup) and from 95.6% to 97.7% on LIBERO, highlighting its effectiveness even under generic robot-manipulation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。