用轨迹抽象表示微调视觉语言模型，提升机器人动作成功判断能力
MotIF: Motion Instruction Fine-tuning
- 通过关键点轨迹叠加图实现运动轨迹的语义抽象表示
- 在MotIF-1K数据集上精度超现有模型2倍，召回率提升56.1%
- 适用于未见过的动作、任务和环境，可指导机器人规划与决策
许多机器人任务的成功需依赖对完整运动过程的观察，而不仅是最终状态。例如梳理头发需重复符合发质和轮廓的刷动。现有方法多采用现成的视觉语言模型（VLM）作为成功判别器，但这类模型仅训练于单帧图像，难以捕捉轨迹信息；即便输入多帧聚合数据，也因缺乏机器人运动数据而表现不佳。本文提出运动指令微调（MotIF），利用关键点轨迹叠加于终帧图像生成的抽象表示，对VLM进行微调，使其能语义化地理解机器人行为。我们构建了包含653个人类和369个机器人演示的MotIF-1K数据集，涵盖13类任务。实验表明，该模型在精度上至少优于当前最优VLM两倍，召回率提升56.1%，并可在未见动作、任务和环境中泛化。此外，我们展示了其在优化机器人规划、终止执行及按任务描述排序轨迹中的实际应用。
原文摘要 · Abstract (English)
While success in many robotics tasks can be determined by only observing the final state and how it differs from the initial state - e.g., if an apple is picked up - many tasks require observing the full motion of the robot to correctly determine success. For example, brushing hair requires repeated strokes that correspond to the contours and type of hair. Prior works often use off-the-shelf vision-language models (VLMs) as success detectors; however, when success depends on the full trajectory, VLMs struggle to make correct judgments for two reasons. First, modern VLMs are trained only on single frames, and cannot capture changes over a full trajectory. Second, even if we provide state-of-the-art VLMs with an aggregate input of multiple frames, they still fail to detect success due to a lack of robot data. Our key idea is to fine-tune VLMs using abstract representations that are able to capture trajectory-level information such as the path the robot takes by overlaying keypoint trajectories on the final image. We propose motion instruction fine-tuning (MotIF), a method that fine-tunes VLMs using the aforementioned abstract representations to semantically ground the robot's behavior in the environment. To benchmark and fine-tune VLMs for robotic motion understanding, we introduce the MotIF-1K dataset containing 653 human and 369 robot demonstrations across 13 task categories. MotIF assesses the success of robot motion given the image observation of the trajectory, task instruction, and motion description. Our model significantly outperforms state-of-the-art VLMs by at least twice in precision and 56.1% in recall, generalizing across unseen motions, tasks, and environments. Finally, we demonstrate practical applications of MotIF in refining and terminating robot planning, and ranking trajectories on how they align with task and motion descriptions. Project page: https://motif-1k.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。