用大模型让机器人理解自然语言指令,实现3D场景中开放词汇的物体功能识别。
3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds
- 将功能检测改为根据自然语言指令推理生成掩码的新任务
- 在开放词汇任务上比传统方法提升约8%的分割准确率
- 适合需要理解复杂指令的机器人场景应用
3D功能检测是机器人任务中的关键挑战。现有方法多依赖预定义标签进行语义分割,难以理解自然语言,泛化能力受限。为此,本文提出新任务——指令推理功能分割(IRAS),即根据自然语言查询生成功能区域掩码,摆脱固定类别限制。为此设计3D-AffordanceLLM框架,引入大语言模型并定制解码器生成掩码,实现开放世界的功能推理。针对3D功能数据稀缺问题,采用多阶段训练:先通过新颖的指代物体部件分割(ROPS)预训练,获得物体-部件级识别与分割能力;再在IRAS任务上微调,获得功能推理能力。实验表明,该方法在开放词汇功能检测任务上实现约8%的mIoU提升。
原文摘要 · Abstract (English)
3D Affordance detection is a challenging problem with broad applications on various robotic tasks. Existing methods typically formulate the detection paradigm as a label-based semantic segmentation task. This paradigm relies on predefined labels and lacks the ability to comprehend complex natural language, resulting in limited generalization in open-world scene. To address these limitations, we reformulate the traditional affordance detection paradigm into \textit{Instruction Reasoning Affordance Segmentation} (IRAS) task. This task is designed to output a affordance mask region given a query reasoning text, which avoids fixed categories of input labels. We accordingly propose the \textit{3D-AffordanceLLM} (3D-ADLLM), a framework designed for reasoning affordance detection in 3D open-scene. Specifically, 3D-ADLLM introduces large language models (LLMs) to 3D affordance perception with a custom-designed decoder for generating affordance masks, thus achieving open-world reasoning affordance detection. In addition, given the scarcity of 3D affordance datasets for training large models, we seek to extract knowledge from general segmentation data and transfer it to affordance detection. Thus, we propose a multi-stage training strategy that begins with a novel pre-training task, i.e., \textit{Referring Object Part Segmentation}~(ROPS). This stage is designed to equip the model with general recognition and segmentation capabilities at the object-part level. Then followed by fine-tuning with the IRAS task, 3D-ADLLM obtains the reasoning ability for affordance detection. In summary, 3D-ADLLM leverages the rich world knowledge and human-object interaction reasoning ability of LLMs, achieving approximately an 8\% improvement in mIoU on open-vocabulary affordance detection tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。