让机器人从八分钟前的记忆中精准调用关键信息,实现长时间自主操作。
Memory Retrieval in Visuomotor Policies for Long-Horizon Robot Control

- 用视觉语言模型先验指导记忆检索,避免错误关联
- 采用稀疏注意力机制,仅提取历史中最相关片段
- 适合需要长期记忆的家用机器人任务
在部分可观测环境(如家庭)中运行的通用机器人需具备记忆能力以实现自主性,能够回忆物体位置、已完成任务及设备开启时间等信息。为实现这一目标,本文提出HALO,一种基于注意力机制的记忆检索型视觉-运动策略。该方法通过将视觉-语言模型(VLM)的先验知识注入策略中,利用演示轨迹生成记忆相关的问答对,并联合训练视频问答任务,引导记忆检索聚焦于任务相关的信息,抑制因数据驱动导致的虚假关联。同时,通过稀疏注意力机制限制记忆检索范围,仅关注历史中最相关的部分,有效缓解因预测误差累积引发的模型漂移与级联故障。实验表明,该方法可稳定支持长达八分钟的历史经验检索,在长时序机器人控制中表现更可靠。
原文摘要 · Abstract (English)
General-purpose robots operating in partially observable environments, such as homes, require memory to support autonomy. They must recall diverse information from the past, such as where objects were placed, which tasks a human partner has completed, and when an appliance was turned on. Achieving this versatility requires a general memory retrieval mechanism. Transformer architectures that use attention over long contexts for memory retrieval provide a promising approach, as they learn retrieval from data rather than relying on task-specific or hand-designed rules. However, directly incorporating them into imitation learning from offline data introduces two key challenges: (1) the policy may learn spurious correlations between past information and predicted actions, and (2) errors accumulate in memory due to prediction inaccuracies and their compounding interactions with the environment, causing model drift and cascading failures. To address both challenges, we introduce HALO, a visuomotor policy with an attention-based memory retrieval mechanism for long-horizon control. First, to suppress spurious correlations, HALO distills vision-language model (VLM) priors into the policy. It generates memory-dependent question--answer pairs from demonstration trajectories and trains jointly with a video question--answering objective, steering retrieval toward task-relevant information. Second, to reduce the impact of accumulated errors in memory during closed-loop control, HALO uses sparse attention that restricts retrieval to only the most relevant parts of the history. Together, these components enable more reliable long-horizon control by guiding the policy to retrieve task-relevant information from up to eight minutes of past experience. Project website: https://robin-lab.cs.utexas.edu/HALO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。