arXiv:2606.08542cs.ROcs.AI2026-06

让视觉模型学会从失败操作中读出隐藏条件,提升机器人任务推理准确率。

When Video Misreads: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QA

论文配图:When Video Misreads: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QA
图 1 · 摘自论文原文
  • 用闭环蒸馏生成自然语言提示,指导模型理解失败操作中的隐含条件
  • 在3个仿真和2个真实机器人任务上,准确率提升0.38至0.47
  • 只需一个提示即可实现零样本程序化分类器,适合机器人自主决策场景

探索性操作常将看似失败的尝试转化为下一步的关键证据。例如,机器人拉一个锁住的抽屉,失败后才意识到需先开锁。这一失败动作揭示了决定最小成功动作链(如[开锁, 拉抽屉])的隐藏前提。我们将其形式化为探索性操作轨迹问答(EMT-QA):给定同步的视频与本体感知数据,预测由探测行为揭示的最小成功动作链。然而,即使最先进的视觉语言模型(VLM)和具身多模态大模型也难以正确解读此类证据:无法从原始视频、原始本体感知或其组合中可靠恢复动作链。我们提出闭环轨迹蒸馏(Closed-Loop Trace Distillation):利用每任务编码代理分析标注训练轨迹,蒸馏出一条关于轨迹的自然语言提示,称为蒸馏阅读启发式(DRH)。推理时无需调用代理,不更新模型权重;仅将原始轨迹与DRH作为提示输入冻结的VLM。在三个模拟器和两个真实机器人任务中,DRH使动作链准确率较最佳原始模态基线提升0.38至0.47。同一DRH还可作为单样本程序化分类器的唯一规范,与提示的VLM匹配。

原文摘要 · Abstract (English)

Exploratory manipulation often turns an apparent failed attempt into the key evidence for what to do next. For example, a robot pulls a locked cabinet drawer, fails, and only succeeds after opening the lock. The failed pull reveals a latent precondition (the drawer is locked) that determines the minimal-success action chain (the fewest actions that complete the task), here [lock-open, drawer-pull]. Correctly reading this trace is therefore the prerequisite for recovering that chain. We formalize this setting as Exploratory Manipulation Trace QA (EMT-QA): given synchronized video and proprioception from an exploratory trace, predict the minimal-success action chain under the latent precondition revealed by the probe. However, even state-of-the-art VLMs and embodied multimodal LLMs misread this evidence: they do not reliably recover the chain from raw video, raw proprioception, or their combination. We introduce Closed-Loop Trace Distillation, a pipeline that uses a per-task coding agent to inspect labeled training traces and distill a one-line natural-language prompt over the trace, which we call the Distilled Reading Heuristic (DRH). At inference, no agent is invoked and no model weights are updated; a frozen VLM receives the raw trace plus the DRH as a prompt entry. Across three simulator and two real-robot tasks, the DRH improves chain accuracy by +0.38 to +0.47 over the best raw-modality baseline. The same DRH also serves as the sole specification for one-shot programmatic classifiers that match the prompted VLM.

机器人视觉语言模型动作推理提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。