用手绘符号图指导机器人操作,无需说话也能精准执行复杂任务。
Robotic Visual Instruction
- 通过手绘箭头、圆圈等符号表达空间时间指令
- 真实场景中未见任务成功率高达87.5%
- 适合需要静音环境或低算力设备的机器人应用
近期,自然语言成为人机交互的主要方式,但其缺乏空间精度,导致任务定义模糊且冗长。在图书馆、医院等需保持安静的场所,语音交流也不合适。为此,我们提出机器人视觉指令(RoVI),一种以物体为中心的手绘符号化表达新范式,将时空信息编码为可读的2D草图,利用箭头、圆圈、颜色和数字指导3D机器人操作。为使机器人更好理解并生成精确动作,我们构建了视觉指令具身工作流(VIEW),该流程借助视觉语言模型(VLMs)解析草图输入,通过关键点提取从2D像素空间解码空间与时间约束,并转化为可执行的3D动作序列。我们还构建了一个包含1.5万条实例的专用数据集,用于微调小型VLMs,使其适用于边缘部署。在真实与模拟环境中对11项新任务进行严格验证,结果显示该方法具备显著泛化能力。值得注意的是,在涉及多步操作、干扰和轨迹跟随要求的真实场景中,VIEW实现了87.5%的成功率。
原文摘要 · Abstract (English)
Recently, natural language has been the primary medium for human-robot interaction. However, its inherent lack of spatial precision introduces challenges for robotic task definition such as ambiguity and verbosity. Moreover, in some public settings where quiet is required, such as libraries or hospitals, verbal communication with robots is inappropriate. To address these limitations, we introduce the Robotic Visual Instruction (RoVI), a novel paradigm to guide robotic tasks through an object-centric, hand-drawn symbolic representation. RoVI effectively encodes spatial-temporal information into human-interpretable visual instructions through 2D sketches, utilizing arrows, circles, colors, and numbers to direct 3D robotic manipulation. To enable robots to understand RoVI better and generate precise actions based on RoVI, we present Visual Instruction Embodied Workflow (VIEW), a pipeline formulated for RoVI-conditioned policies. This approach leverages Vision-Language Models (VLMs) to interpret RoVI inputs, decode spatial and temporal constraints from 2D pixel space via keypoint extraction, and then transform them into executable 3D action sequences. We additionally curate a specialized dataset of 15K instances to fine-tune small VLMs for edge deployment,enabling them to effectively learn RoVI capabilities. Our approach is rigorously validated across 11 novel tasks in both real and simulated environments, demonstrating significant generalization capability. Notably, VIEW achieves an 87.5% success rate in real-world scenarios involving unseen tasks that feature multi-step actions, with disturbances, and trajectory-following requirements. Project website: https://robotic-visual-instruction.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。