用3D点引导让机器人零样本抓取，速度快过20秒
Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping
- 基于多模态大模型生成3D动作点，实现任务驱动的精准定位
- 在未见环境中完成自然语言描述的抓取，响应时间低于20秒
- 适合需要快速部署、无需训练的机器人操作场景
我们提出Point2Act，直接从上下文描述的任务中检索相关的3D动作点，利用多模态大语言模型（MLLMs）。基础模型使得通用机器人能够零样本地在未见过的环境中根据自然语言描述执行任务。尽管大规模图像与语言数据集提供的语义理解可支持2D图像中的上下文感知，但其高维特征常导致模糊的2D区域，难以精确定位3D动作位置。我们提出的3D相关性场跳过高维特征，改用轻量级2D点级引导，针对性地适配特定任务的动作需求。多视角聚合有效补偿由遮挡或语言描述固有语义不确定性带来的几何错位。输出区域高度局部化，能推理出精细的3D空间上下文，并实时用于场景重建后的物理动作定位。完整的端到端流程包括采集、MLLM查询、3D重建和抓取姿态提取，在20秒内生成空间对齐的响应，适用于实际操作任务。
原文摘要 · Abstract (English)
We propose Point2Act, which directly retrieves the 3D action point relevant to a contextually described task, leveraging Multimodal Large Language Models (MLLMs). Foundation models opened the possibility for generalist robots that can perform a zero-shot task following natural language descriptions within an unseen environment. While the semantics obtained from large-scale image and language datasets provide contextual understanding in 2D images, the rich yet nuanced features deduce blurry 2D regions and struggle to find precise 3D locations for actions. Our proposed 3D relevancy fields bypass the high-dimensional features and instead efficiently imbue lightweight 2D point-level guidance tailored to the task-specific action. The multi-view aggregation effectively compensates for misalignments due to geometric ambiguities, such as occlusion, or semantic uncertainties inherent in the language descriptions. The output region is highly localized, reasoning fine-grained 3D spatial context that can directly transfer to an explicit position for physical action at the on-the-fly reconstruction of the scene. Our full-stack pipeline, which includes capturing, MLLM querying, 3D reconstruction, and grasp pose extraction, generates spatially grounded responses in under 20 seconds, facilitating practical manipulation tasks. Project page: https://sangminkim-99.github.io/point2act/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。