提出新型像素感知变换器,提升RGB-D场景分割精度
DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentation
- 设计跨模态交互模块,分离特定与共享特征以实现像素级对齐
- 在SUN RGB-D和NYUDv2上分别达54.28%和59.95% mIoU
- 适合做室内场景理解的视觉与机器人研究者参考
室内语义分割是计算机视觉与机器人学的基础任务,支撑自主导航、增强现实及智能环境等应用。尽管RGB-D融合可利用外观与几何信息互补,现有方法多依赖计算量大的交叉注意力机制,且对模态内与模态间特征关系建模不足,导致特征对齐不精准、判别性表征有限。为此,我们提出DiffPixelFormer,一种用于RGB-D室内场景分割的差分像素感知变换器,同时增强模态内表征并建模模态间交互。核心为模态内-模态间交互块(IIMIB),通过自注意力捕获模态内长程依赖,并利用差分-共享跨模态模块(DSIM)分离模态特有与共享线索,实现细粒度像素级跨模态对齐。此外,动态融合策略根据场景特性平衡模态贡献,充分挖掘RGB-D信息。在SUN RGB-D与NYUDv2基准上的大量实验表明,DiffPixelFormer-L在两项任务上分别取得54.28%与59.95%的mIoU,优于DFormer-L 1.78%与2.75%。代码已开源。
原文摘要 · Abstract (English)
Indoor semantic segmentation is fundamental to computer vision and robotics, supporting applications such as autonomous navigation, augmented reality, and smart environments. Although RGB-D fusion leverages complementary appearance and geometric cues, existing methods often depend on computationally intensive cross-attention mechanisms and insufficiently model intra- and inter-modal feature relationships, resulting in imprecise feature alignment and limited discriminative representation. To address these challenges, we propose DiffPixelFormer, a differential pixel-aware Transformer for RGB-D indoor scene segmentation that simultaneously enhances intra-modal representations and models inter-modal interactions. At its core, the Intra-Inter Modal Interaction Block (IIMIB) captures intra-modal long-range dependencies via self-attention and models inter-modal interactions with the Differential-Shared Inter-Modal (DSIM) module to disentangle modality-specific and shared cues, enabling fine-grained, pixel-level cross-modal alignment. Furthermore, a dynamic fusion strategy balances modality contributions and fully exploits RGB-D information according to scene characteristics. Extensive experiments on the SUN RGB-D and NYUDv2 benchmarks demonstrate that DiffPixelFormer-L achieves mIoU scores of 54.28% and 59.95%, outperforming DFormer-L by 1.78% and 2.75%, respectively. Code is available at https://github.com/gongyan1/DiffPixelFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。