端到端联合重建与实例分割，从多视角图像直接输出3D物体结构。
SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images
- 用对象查询融合几何特征，实现视觉-几何联合建模。
- 在ScanNetv2和ScanNet200上达到最优性能，优于现有联合与RGB-D方法。
- 提出帧级注意力对齐策略，提升注意力聚焦，无额外推理开销。
3D实例分割方法通常依赖高质量点云或带姿态的RGB-D扫描，需复杂多阶段处理流程，且对重建噪声敏感。尽管近期前馈式变换器已革新多视角3D重建，但其仍与高层语义理解分离。本文提出SegVGGT，一个统一的端到端框架，直接从多视角RGB图像同时完成前馈式3D重建与实例分割。通过引入与多层级几何特征交互的对象查询，将实例识别深度融入视觉几何基变换器。为解决大量全局图像标记导致的严重注意力分散问题，提出帧级注意力分布对齐（FADA）策略，训练时显式引导对象查询关注相关帧，提供结构化监督且不增加推理开销。大量实验表明，SegVGGT在ScanNetv2和ScanNet200上达到当前最佳性能，超越近期联合模型及基于RGB-D的方法，且在ScanNet++上展现出强泛化能力。
原文摘要 · Abstract (English)
3D instance segmentation methods typically rely on high-quality point clouds or posed RGB-D scans, requiring complex multi-stage processing pipelines, and are highly sensitive to reconstruction noise. While recent feed-forward transformers have revolutionized multi-view 3D reconstruction, they remain decoupled from high-level semantic understanding. In this work, we present SegVGGT, a unified end-to-end framework that simultaneously performs feed-forward 3D reconstruction and instance segmentation directly from multi-view RGB images. By introducing object queries that interact with multi-level geometric features, our method deeply integrates instance identification into the visual geometry grounded transformer. To address the severe attention dispersion problem caused by the massive number of global image tokens, we propose the Frame-level Attention Distribution Alignment (FADA) strategy. FADA explicitly guides object queries to attend to instance-relevant frames during training, providing structured supervision without extra inference overhead. Extensive experiments demonstrate that SegVGGT achieves the state-of-the-art performance on ScanNetv2 and ScanNet200, outperforming both recent joint models and RGB-D-based approaches, while exhibiting strong generalization capabilities on ScanNet++.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。