提出偏移学习机制,提升轻量分割模型的特征对齐效果。
Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignment
- 通过双分支结构学习特征与类别偏移,动态优化空间与类别特征。
- 在ADE20K上使SegFormer-B0等模型提升2.7%以上mIoU,仅增0.1-0.2M参数。
- 可无缝集成至现有轻量模型,适合边缘设备部署场景。
语义分割是实现像素级场景理解的核心任务,但在资源受限设备上部署需高效架构。现有方法虽通过轻量化设计实现实时推理,但存在固有缺陷:由逐像素分类范式导致的类别表征与图像特征错位。实验分析表明,该范式在高效场景下引入了苛刻假设——同一类别的图像像素特征在不同图像中不应变化。为此,我们提出耦合双分支偏移学习范式,显式学习特征偏移与类别偏移,动态修正类别表征与空间图像特征。基于此,构建高效分割网络OffSeg。该偏移学习机制可无修改地应用于现有方法。在ADE20K、Cityscapes、COCO-Stuff-164K和Pascal Context四个数据集上的大量实验表明,其性能持续提升且参数开销极低。例如,在ADE20K上,该方法使SegFormer-B0、SegNeXt-T、Mask2Former-Tiny的mIoU分别提升2.7%、1.9%、2.6%,仅需增加0.1–0.2M额外参数。
原文摘要 · Abstract (English)
Semantic segmentation is fundamental to vision systems requiring pixel-level scene understanding, yet deploying it on resource-constrained devices demands efficient architectures. Although existing methods achieve real-time inference through lightweight designs, we reveal their inherent limitation: misalignment between class representations and image features caused by a per-pixel classification paradigm. With experimental analysis, we find that this paradigm results in a highly challenging assumption for efficient scenarios: Image pixel features should not vary for the same category in different images. To address this dilemma, we propose a coupled dual-branch offset learning paradigm that explicitly learns feature and class offsets to dynamically refine both class representations and spatial image features. Based on the proposed paradigm, we construct an efficient semantic segmentation network, OffSeg. Notably, the offset learning paradigm can be adopted to existing methods with no additional architectural changes. Extensive experiments on four datasets, including ADE20K, Cityscapes, COCO-Stuff-164K, and Pascal Context, demonstrate consistent improvements with negligible parameters. For instance, on the ADE20K dataset, our proposed offset learning paradigm improves SegFormer-B0, SegNeXt-T, and Mask2Former-Tiny by 2.7%, 1.9%, and 2.6% mIoU, respectively, with only 0.1-0.2M additional parameters required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。