arXiv:2607.19765cs.CV2026-07中稿 · ECCV

用视图合成模型传播语义标签,实现无需3D重建的多视角全景分割。

Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

论文配图:Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation
图 1 · 摘自论文原文
  • 将全景标签编码为二值通道输入视图合成模型,实现跨视角传播。
  • 在ScanNet上分割质量媲美需3D重建的方法,新视图合成性能提升7 dB以上。
  • 无需微调即可跨数据集迁移,适合快速部署于新场景的3D理解任务。

大型视图合成模型通过跨视图注意力实现新视角生成,无需显式3D表示,近期研究发现其仅凭RGB监督即可学习准确的空间对应关系。我们观察到这种对应关系不仅适用于外观,还能推广至非真实感信号——当二值编码的全景标签输入模型时,其空间结构可一致地传播至新视图。基于此,我们首次将大规模视图合成模型拓展至3D场景理解任务。提出一种全景分割流程:复用冻结的视图合成模型,将输入视图的全景标签编码为二值通道后输入,直接生成目标视图的分割结果,无需3D重建或对视图合成模型进行分割特化训练。在ScanNet上,本方法分割质量与依赖高斯表示的重建方法相当,且新视图合成性能优于后者7 dB以上;标签传播能力跨数据集迁移,在Replica上无需微调即超越现有方法。

原文摘要 · Abstract (English)

Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We observe that this correspondence generalizes beyond appearance. When non-photorealistic signals such as binary encoded panoptic labels are passed through the model, they are propagated to novel views with consistent spatial structure. These results indicate that the correspondence learned for RGB view synthesis can also propagate view-independent per-pixel labels. From this observation, we present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. We propose a panoptic segmentation pipeline that reuses a frozen view synthesis model to propagate panoptic labels from input views to novel views, without 3D reconstruction or any segmentation-specific training of the view synthesis model. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation. On ScanNet, our method achieves segmentation quality on par with Gaussian based approaches requiring explicit 3D reconstruction, while outperforming them in novel view synthesis by more than 7 dB. The label propagation also transfers across datasets, surpassing these approaches on Replica without any fine-tuning.

全景分割视图合成3D理解标签传播

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。