用视觉语言模型从单目图像预测3D物体位置，让机器人更懂人类指令。
Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction
- 在单目图像上结合语言指令和机器人状态，推断物体3D坐标。
- 测试集上中位误差13毫米，比基线提升五倍。
- 适合需要精准定位的机器人交互任务，如家庭服务。
预训练通用视觉语言模型（VLM）凭借其丰富的世界知识和2D目标检测能力，有望提升人机交互的自然性。然而，针对3D坐标检测任务的VLM仍较为稀缺。本文研究了在腕戴摄像头单目RGB图像、自然语言输入和机器人状态条件下，利用VLM返回3D物体位置的能力。我们构建并整理了一个包含超过10万张图像的异构数据集，并采用QLoRA微调带有自定义回归头的VLM。通过引入条件路由机制，模型在保持处理通用视觉查询能力的同时，新增了专用的3D位置估计功能。实验结果表明，该模型在测试集上具有稳健的预测性能，中位平均绝对误差（MAE）为13毫米，相比未微调的简单基线提升五倍。约25%的预测结果达到机器人可交互的可接受范围。
原文摘要 · Abstract (English)
Pre-trained general-purpose Vision-Language Models (VLM) hold the potential to enhance intuitive human-machine interactions due to their rich world knowledge and 2D object detection capabilities. However, VLMs for 3D coordinates detection tasks are rare. In this work, we investigate interactive abilities of VLMs by returning 3D object positions given a monocular RGB image from a wrist-mounted camera, natural language input, and robot states. We collected and curated a heterogeneous dataset of more than 100,000 images and finetuned a VLM using QLoRA with a custom regression head. By implementing conditional routing, our model maintains its ability to process general visual queries while adding specialized 3D position estimation capabilities. Our results demonstrate robust predictive performance with a median MAE of 13 mm on the test set and a five-fold improvement over a simpler baseline without finetuning. In about 25% of the cases, predictions are within a range considered acceptable for the robot to interact with objects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。