用新型深度表示与动态融合提升全景语义分割精度
REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion
- 提出基于柱坐标系的REL深度表示，融合深度、仰角和方位角
- 在斯坦福2D3D数据集上平均mIoU提升2.35%，方差降低约70%
- 适合做全景视觉理解的科研与工业应用，尤其关注鲁棒性
全景语义分割（PASS）旨在通过超广视角实现完整场景感知。现有方法多依赖球面几何与RGB输入或原始/ HHA格式深度信息，未能充分挖掘全景图像几何特性。为此，本文提出REL-SF4PASS，采用基于柱坐标系的REL深度表示与球面动态多模态融合（SMMF）。REL由校正深度、增益仰角和侧向方位角构成，全面表达柱坐标系下的三维空间及表面法向。SMMF根据不同全景区域特点采用差异化融合策略，缓解圆柱面在等距矩形投影中的展开断裂问题。实验表明，该方法在斯坦福2D3D全景数据集上显著提升性能与鲁棒性：所有三折平均mIoU提升2.35%，面对三维扰动时性能方差减少约70%。
原文摘要 · Abstract (English)
As an important and challenging problem in computer vision, Panoramic Semantic Segmentation (PASS) aims to give complete scene perception based on an ultra-wide angle of view. Most PASS methods often focus on spherical geometry with RGB input or using the depth information in original or HHA format, which does not make full use of panoramic image geometry. To address these shortcomings, we propose REL-SF4PASS with our REL depth representation based on cylindrical coordinate and Spherical-dynamic Multi-Modal Fusion SMMF. REL is made up of Rectified Depth, Elevation-Gained Vertical Inclination Angle, and Lateral Orientation Angle, which fully represents 3D space in cylindrical coordinate style and the surface normal direction. SMMF aims to ensure the diversity of fusion for different panoramic image regions and reduce the breakage of cylinder side surface expansion in ERP projection, which uses different fusion strategies to match the different regions in panoramic images. Experimental results show that REL-SF4PASS considerably improves performance and robustness on popular benchmark, Stanford2D3D Panoramic datasets. It gains 2.35% average mIoU improvement on all 3 folds and reduces the performance variance by approximately 70% when facing 3D disturbance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。