通过关键帧链建模长时序依赖,提升机器人操作泛化能力
Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chaining
- 自动提取关键历史帧,构建可区分的状态嵌入空间
- 动态检索与当前阶段相关的过去帧,提升任务成功率至87.3%
- 适合需要长时序记忆的复杂机器人操作任务
现有视觉-语言-动作(VLA)模型因过度依赖即时观测,在长时程任务上泛化能力有限。尽管近期研究引入检索机制或扩展上下文窗口处理流程任务,但仍难以捕捉非马尔可夫依赖关系——即最优动作仅依赖特定历史状态而非当前观测。为此,我们提出关键帧链接式VLA框架,通过自动关键帧选择器学习判别性嵌入空间,有效识别状态转换。设计进度感知查询机制,动态根据时间相关性检索历史帧。选定的关键帧以交错视觉标记形式融入VLA,显式建模长时序上下文。最后,基于ManiSkill仿真器构建四类非马尔可夫操作任务,评估任务成功率。实验表明,该方法在长时序依赖任务中表现优异。代码已公开于https://github.com/cytoplastm/KC-VLA。
原文摘要 · Abstract (English)
Existing Vision-Language-Action (VLA) models often struggle to generalize to long-horizon tasks due to their heavy reliance on immediate observations. While recent studies incorporate retrieval mechanisms or extend context windows to handle procedural tasks, they often struggle to capture Non-Markovian dependencies, where optimal actions rely solely on specific past states rather than the current observation. To address this, we introduce Keyframe-Chaining VLA, a framework that extracts and links key historical frames to model long-horizon dependencies. Specifically, we propose an automatic keyframe selector that learns a discriminative embedding space, effectively identifying distinct state transitions. To capture task-critical information, we design a progress-aware query mechanism that dynamically retrieves historical frames based on their temporal relevance to the current execution phase. These selected keyframes are integrated into the VLA as interleaved visual tokens, explicitly grounding the policy in the long-horizon temporal context. Finally, we introduce a suite of four Non-Markovian manipulation tasks built upon the ManiSkill simulator to measure task success rates. Experimental results demonstrate that our method achieves superior performance, effectively tackling robot manipulation tasks characterized by long-horizon temporal dependencies. Code is available at https://github.com/cytoplastm/KC-VLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。