arXiv:2511.10560cs.CV2025-11被引 18

让视觉模型同时利用多种几何信息,提升3D理解能力。

OmniVGGT: Omni-Modality Driven Visual Geometry Grounded Transformer

  • 用GeoAdapter将深度、相机参数等几何信息逐步注入基础模型。
  • 支持任意数量的几何输入,测试时无需固定模态组合。
  • 适合需要多模态感知的机器人与3D视觉任务

通用3D基础模型正推动视觉任务统一,但多数仅依赖RGB输入,忽略相机内参、位姿和深度图等几何线索。为此,我们提出OmniVGGT框架,可在训练和推理阶段有效利用任意数量的辅助几何模态。其核心是GeoAdapter,通过零初始化卷积将深度与相机内外参编码进空间基础模型,不破坏原有表征空间,优化稳定且计算开销极小,推理速度与VGGT相当。此外,采用随机采样模态子集的多模态融合策略,使模型在测试时可灵活处理任意模态组合,增强对空间特征的鲁棒性。在单目/多视图深度估计、多视图立体匹配和相机位姿估计上,OmniVGGT优于已有含辅助输入方法,甚至在仅用RGB输入时也达当前最优。进一步集成至视觉-语言-动作(VLA)模型后,该增强版模型在主流基准上超越基于点云的基线,并在机器人任务中持续受益于可用几何信息。

原文摘要 · Abstract (English)

General 3D foundation models have started to lead the trend of unifying diverse vision tasks, yet most assume RGB-only inputs and ignore readily available geometric cues (e.g., camera intrinsics, poses, and depth maps). To address this issue, we introduce OmniVGGT, a novel framework that can effectively benefit from an arbitrary number of auxiliary geometric modalities during both training and inference. In our framework, a GeoAdapter is proposed to encode depth and camera intrinsics/extrinsics into a spatial foundation model. It employs zero-initialized convolutions to progressively inject geometric information without disrupting the foundation model's representation space. This design ensures stable optimization with negligible overhead, maintaining inference speed comparable to VGGT even with multiple additional inputs. Additionally, a stochastic multimodal fusion regimen is proposed, which randomly samples modality subsets per instance during training. This enables an arbitrary number of modality inputs during testing and promotes learning robust spatial representations instead of overfitting to auxiliary cues. Comprehensive experiments on monocular/multi-view depth estimation, multi-view stereo, and camera pose estimation demonstrate that OmniVGGT outperforms prior methods with auxiliary inputs and achieves state-of-the-art results even with RGB-only input. To further highlight its practical utility, we integrated OmniVGGT into vision-language-action (VLA) models. The enhanced VLA model by OmniVGGT not only outperforms the vanilla point-cloud-based baseline on mainstream benchmarks, but also effectively leverages accessible auxiliary inputs to achieve consistent gains on robotic tasks.

3D视觉多模态几何建模机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。