让大模型直接生成机器人可执行的运动轨迹，实现零样本操作。
Language Movement Primitives: Grounding Language Models in Robot Motion
- 用动态运动基元（DMP）参数化运动，让语言模型直接控制机器人动作。
- 在31个真实任务中达65%成功率，远超基线的35%。
- 适合想用自然语言指挥机器人完成新任务的研究者和工程师。
尽管基础模型在通用问题求解方面取得显著进展，但让机器人根据自然语言指令完成新操作任务仍是重大挑战。大型视觉语言模型（VLMs）能处理高维输入，理解场景与语言，并将任务分解为逻辑步骤，却难以将这些步骤转化为具身机器人的实际运动。相反，机器人基础模型虽能输出动作指令，但需领域微调或经验积累才能完成新任务。核心难题在于如何将高层任务推理与底层运动控制相连接。为此，我们提出语言运动基元（LMP），通过动态运动基元（DMP）参数化，将VLM的推理结果语义地映射到连续、稳定且可解释的运动轨迹上。具体而言，VLM解析自然语言指令，设定DMP参数以生成多样化运动。基于此，我们构建了零样本机器人操作流水线，能有效完成桌面操作任务。在31个真实世界操作任务中，LMP达到65%的任务成功率，优于最佳基线的35%。更多视频请见：https://collab.me.vt.edu/lmp
原文摘要 · Abstract (English)
Enabling robots to perform novel manipulation tasks from natural language instructions remains a fundamental challenge in robotics, despite significant progress in generalized problem solving with foundational models. Large vision and language models (VLMs) are capable of processing high-dimensional input data for visual scene and language understanding, as well as decomposing tasks into a sequence of logical steps; however, they struggle to ground those steps in embodied robot motion. On the other hand, robotics foundation models output action commands, but require in-domain fine-tuning or experience before they are able to perform novel tasks successfully. At its core, there still remains the fundamental challenge of connecting abstract task reasoning with low-level motion control. To address this disconnect, we propose Language Movement Primitives (LMPs), a framework that grounds VLM reasoning in Dynamic Movement Primitive (DMP) parameterization. Our key insight is that DMPs provide a small number of interpretable parameters, and VLMs can set these parameters to specify diverse, continuous, and stable trajectories. Put another way: VLMs can reason over free-form natural language task descriptions, and semantically ground their desired motions into DMPs -- bridging the gap between high-level task reasoning and low-level position and velocity control. Building on this combination of VLMs and DMPs, we formulate our LMP pipeline for zero-shot robot manipulation that effectively completes tabletop manipulation problems by generating a sequence of DMP motions. Across 31 real-world manipulation tasks, we show that LMP achieves 65% task success as compared to 35% for the best performing baseline. See videos at our website: https://collab.me.vt.edu/lmp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。