用少视角图像实现快速精准3D目标分割,适合机器人等实时场景。
MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation
- 双分支结构融合语言与稀疏视图几何信息,端到端完成分割。
- 在稀疏视图下达到81.3%的mAP,推理速度超60帧/秒。
- 提出新优化方法缓解稀疏信号导致的梯度稀释问题,适合移动端部署。
现有3D指代表达分割(3DRES)方法依赖密集高质量点云,而真实世界中的机器人、手机等设备仅能获取少量稀疏彩色图像且有严格延迟要求。我们提出多视角3D指代表达分割(MV-3DRES),模型需直接从稀疏多视角图像中恢复场景结构并分割目标物体。传统两阶段方法先重建点云再分割,常因几何质量差、目标区域粗糙或退化且运行缓慢。为此,我们提出多模态视觉几何锚定变换器(MVGGT),一种高效端到端框架,通过双分支设计将语言信息融入稀疏视图几何推理。训练中发现关键优化障碍——前景梯度稀释(FGD),由稀疏3D信号导致监督信号弱。为此,我们引入每视图无目标抑制优化(PVSO),在各视角提供更强更均衡的梯度,实现稳定高效学习。为支持一致评估,我们构建了MVRefer基准,定义标准化设置与指标。实验表明,MVGGT建立了首个强基线,在准确率与推理速度上均优于现有方法。代码已开源。
原文摘要 · Abstract (English)
Most existing 3D referring expression segmentation (3DRES) methods rely on dense, high-quality point clouds, while real-world agents such as robots and mobile phones operate with only a few sparse RGB views and strict latency constraints. We introduce Multi-view 3D Referring Expression Segmentation (MV-3DRES), where the model must recover scene structure and segment the referred object directly from sparse multi-view images. Traditional two-stage pipelines, which first reconstruct a point cloud and then perform segmentation, often yield low-quality geometry, produce coarse or degraded target regions, and run slowly. We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an efficient end-to-end framework that integrates language information into sparse-view geometric reasoning through a dual-branch design. Training in this setting exposes a critical optimization barrier, termed Foreground Gradient Dilution (FGD), where sparse 3D signals lead to weak supervision. To resolve this, we introduce Per-view No-target Suppression Optimization (PVSO), which provides stronger and more balanced gradients across views, enabling stable and efficient learning. To support consistent evaluation, we build MVRefer, a benchmark that defines standardized settings and metrics for MV-3DRES. Experiments show that MVGGT establishes the first strong baseline and achieves both high accuracy and fast inference, outperforming existing alternatives. The code is available at https://mvggt.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。