让机器人同时理解文字和图像,用交替的文本与视觉线索规划长任务。
Thinking in Text and Images: Interleaved Vision--Language Reasoning Traces for Long-Horizon Robot Manipulation

- 用文本子目标与视觉关键帧交替生成全局规划轨迹
- 在LIBERO数据集上达92.4%长任务成功率,优于纯文本或纯视觉方案
- 适合需要多步推理与空间理解的复杂机器人操作场景
长时序机器人操作需兼具逻辑连贯性与几何准确性。现有视觉-语言-动作策略通常将规划隐藏于隐状态,或仅暴露单一模态:纯文本思维链保留因果顺序但忽略空间约束,视觉预测提供几何线索却常局限于局部且语义不足。本文提出交织式视觉-语言推理(IVLR),以\trace{}为显式中间表示,贯穿全任务周期交替输出文本子目标与视觉关键帧。测试时,单个原生多模态Transformer从初始观测与指令自动生成全局语义-几何轨迹,缓存后用于闭环动作解码,条件依赖轨迹、指令与当前观测。因标准机器人数据集缺乏此类轨迹,我们通过时间分段演示并用视觉-语言模型标注各阶段构建伪监督信号。在模拟长时序操作基准与视觉分布偏移测试中,该方法在LIBERO上平均成功率达95.5%,其中LIBERO-Long为92.4%;在SimplerEnv-WidowX上总体成功率为59.4%。消融实验表明双模态缺一不可:无轨迹时LIBERO-Long成功率降至37.7%;纯文本与纯视觉轨迹分别达62.0%与68.4%,而完整交织轨迹达92.4%。抗扰动与轨迹遮蔽测试显示轨迹可容忍局部损坏与适度执行漂移,但在全局计划过时或错误时仍受限。
原文摘要 · Abstract (English)
Long-horizon robotic manipulation requires plans that are both logically coherent and geometrically grounded. Existing Vision-Language-Action policies usually hide planning in latent states or expose only one modality: text-only chain-of-thought encodes causal order but misses spatial constraints, while visual prediction provides geometric cues but often remains local and semantically underconstrained. We introduce Interleaved Vision--Language Reasoning (IVLR), a policy framework built around \trace{}, an explicit intermediate representation that alternates textual subgoals with visual keyframes over the full task horizon. At test time, a single native multimodal transformer self-generates this global semantic-geometric trace from the initial observation and instruction, caches it, and conditions a closed-loop action decoder on the trace, original instruction, and current observation. Because standard robot datasets lack such traces, we construct pseudo-supervision by temporally segmenting demonstrations and captioning each stage with a vision-language model. Across simulated benchmarks for long-horizon manipulation and visual distribution shift, \method{} reaches 95.5\% average success on LIBERO, including 92.4\% on LIBERO-Long, and 59.4\% overall success on SimplerEnv-WidowX. Ablations show that both modalities are necessary: without traces, LIBERO-Long success drops to 37.7\%; text-only and vision-only traces reach 62.0\% and 68.4\%, while the full interleaved trace reaches 92.4\%. Stress tests with execution perturbations and masked trace content show moderate degradation, suggesting that the trace can tolerate local corruption and moderate execution drift, but remains limited under stale or incorrect global plans.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。