无需训练即可让机器人听懂指令完成复杂操作
Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation
- 分三阶段处理：视觉感知→语义理解→任务执行
- 零样本任务成功率62%，无需特定领域训练
- 适合需要快速部署的智能机器人场景
本文提出一种无需训练的模块化框架，实现半结构化环境中零样本、语言引导的机器人操作。该架构将高层推理与底层运动学结合，将视觉-动作流程分解为三个阶段：视觉感知、语义解析和任务执行。为克服标准视觉-语言模型（VLM）存在的空间模糊和语义幻觉问题，感知模块采用FastSAM与一组标记（SoM）提示，动态生成可定位的字母数字视觉锚点。同一基础模型随后仅作为大语言模型（LLM）运行，充当语义路由，将自由形式的人类指令转化为可验证、可重构的配置。这些配置由任务编排器动态解析为MoveIt任务构造器（MTC），生成无碰撞轨迹。在两个零样本实验设置中评估：开放世界序列操作和密集关系空间推理，整体端到端任务成功率达62%，证明其可在不进行领域特定训练或手动坐标编程的情况下，可靠执行复杂物理动作。
原文摘要 · Abstract (English)
This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。