用深度图做几何先验，提升复杂光照下语义分割精度
DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation
- 深度图不进网络，直接作为几何先验指导注意力分配
- 在多个基准上达到领先性能，复杂光照下优势显著
- 适合追求高效高精度的视觉理解研究者
场景理解近期进展得益于深度图提供的三维几何信息，尤其在低光和过曝等复杂条件下。现有方法将深度图与RGB图像一起编码，并进行特征融合以提升预测鲁棒性。基于深度图可视为对RGB图像的几何补充这一洞察，本文提出新思路：是否必须像处理RGB图像那样用神经网络显式编码深度信息？为此，我们提出DFormerv2，一种强效的RGBD编码器，其不通过神经网络编码深度，而是直接将深度图作为几何先验使用。目标是从深度图和所有图像块之间的空间距离中提取几何线索，进而作为几何先验分配自注意力权重。大量实验表明，DFormerv2在多个RGBD语义分割基准上表现卓越。代码已开源：https://github.com/VCIP-RGBD/DFormer。
原文摘要 · Abstract (English)
Recent advances in scene understanding benefit a lot from depth maps because of the 3D geometry information, especially in complex conditions (e.g., low light and overexposed). Existing approaches encode depth maps along with RGB images and perform feature fusion between them to enable more robust predictions. Taking into account that depth can be regarded as a geometry supplement for RGB images, a straightforward question arises: Do we really need to explicitly encode depth information with neural networks as done for RGB images? Based on this insight, in this paper, we investigate a new way to learn RGBD feature representations and present DFormerv2, a strong RGBD encoder that explicitly uses depth maps as geometry priors rather than encoding depth information with neural networks. Our goal is to extract the geometry clues from the depth and spatial distances among all the image patch tokens, which will then be used as geometry priors to allocate attention weights in self-attention. Extensive experiments demonstrate that DFormerv2 exhibits exceptional performance in various RGBD semantic segmentation benchmarks. Code is available at: https://github.com/VCIP-RGBD/DFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。