用多样点云增强视觉语言动作模型，提升复杂场景理解能力
Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds
- 将2D图像转换为多源点云，融合3D与2D视觉表示
- 在模拟和真实场景中均显著减少域差异，提升模型鲁棒性
- 适合需要强3D空间感知的机器人控制与具身智能研究
现有视觉-语言-动作（VLA）模型通常以2D图像作为视觉输入，限制了其在复杂场景中的空间理解能力。如何引入3D信息以增强VLA性能？我们开展跨观测空间与视觉表征的初步研究，结果表明：显式地将视觉输入升维为点云，能生成更优的互补表示。针对3D数据稀缺、跨环境差异及深度尺度偏差带来的域差距问题，提出Any3D-VLA。该方法在训练流程中统一仿真器、传感器与模型估计的点云，构建多样化输入，学习与领域无关的3D表示，并与对应2D表示融合。仿真与真实世界实验均验证了Any3D-VLA在性能提升与域差距缓解方面的优势。
原文摘要 · Abstract (English)
Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our project homepage is available at https://xianzhefan.github.io/Any3D-VLA.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。