用图神经网络融合深度与图像信息,提升分割精度与边界清晰度。
Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networks
- 先融合再建图:基于纹理引导几何特征注入,实现晚期融合
- 引入GNN推理补丁关系,有效减少不规则分割块
- 改进投影矩阵生成,解决误分配与局部模糊问题
现有大多数RGB-D语义分割方法聚焦于特征级融合,采用复杂的跨模态、跨尺度融合模块,但易引发特征对齐偏差及分割结果中出现反直觉的不规则区域。受像素-节点-像素流程启发,本文提出:1)在后期融合阶段,以纹理特征为先验引导几何特征注入;2)在融合特征上使用图神经网络(GNN),通过推断补丁间关系缓解不规则块的产生。在3D特征提取阶段,认为传统CNN对深度图效率不足,故将深度图编码为法向图,使CNN更易捕捉物体表面趋势。在投影矩阵生成阶段,发现原流程存在偏差分配(Biased-Assignment)和模糊局部性(Ambiguous-Locality)问题,因此提出:1)采用Kullback-Leibler损失确保重要像素特征不丢失,可视为硬样本挖掘过程;2)在欧氏空间与语义空间中相近的区域赋予更大边权重,综合考虑位置信息。在NYU-DepthV2与SUN RGB-D两个公开数据集上的大量实验表明,该方法能持续提升RGB-D语义分割性能。
原文摘要 · Abstract (English)
Most existing RGB-D semantic segmentation methods focus on the feature level fusion, including complex cross-modality and cross-scale fusion modules. However, these methods may cause misalignment problem in the feature fusion process and counter-intuitive patches in the segmentation results. Inspired by the popular pixel-node-pixel pipeline, we propose to 1) fuse features from two modalities in a late fusion style, during which the geometric feature injection is guided by texture feature prior; 2) employ Graph Neural Networks (GNNs) on the fused feature to alleviate the emergence of irregular patches by inferring patch relationship. At the 3D feature extraction stage, we argue that traditional CNNs are not efficient enough for depth maps. So, we encode depth map into normal map, after which CNNs can easily extract object surface tendencies.At projection matrix generation stage, we find the existence of Biased-Assignment and Ambiguous-Locality issues in the original pipeline. Therefore, we propose to 1) adopt the Kullback-Leibler Loss to ensure no missing important pixel features, which can be viewed as hard pixel mining process; 2) connect regions that are close to each other in the Euclidean space as well as in the semantic space with larger edge weights so that location informations can been considered. Extensive experiments on two public datasets, NYU-DepthV2 and SUN RGB-D, have shown that our approach can consistently boost the performance of RGB-D semantic segmentation task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。