用视觉轨迹引导长程操作,自动修复失败步骤。
Long-Horizon Manipulation via Trace-Conditioned VLA Planning

- 分层管理:用语言+视觉轨迹规划下一步动作
- 实测在仿真和真实机器人上成功率达78%以上
- 无需人工设计恢复逻辑,适合复杂任务场景
长时序操作对视觉-语言-动作(VLA)策略仍具挑战:真实任务多步依赖、易受累积执行误差影响。本文提出LoHo-Manip,一种模块化框架,通过专用任务管理视觉语言模型(VLM),将短时序VLA扩展至长时序指令执行。管理器与执行器解耦,以滚动视野方式调用:根据当前观测,预测一个进度感知的剩余计划,包含(i)显式完成/剩余分割的子任务序列作为轻量语言记忆,以及(ii)视觉轨迹——一个紧凑的2D关键点轨迹提示,指明下一步去向与目标。执行器VLA适配为条件于渲染轨迹,将长时序决策转化为重复局部控制。关键在于每一步预测剩余计划,形成隐式闭环:失败步骤持续出现在后续输出中,轨迹随之更新,实现无需人工设计恢复逻辑或脆弱视觉历史缓冲的自动续行与重规划。大量实验涵盖具身规划、长时推理、轨迹预测及仿真与真实Franka机器人上的端到端操作,均显示在长时序成功率、鲁棒性及分布外泛化能力上显著提升。
原文摘要 · Abstract (English)
Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。