arXiv:2501.19255cs.CV2025-01被引 5

融合卷积与注意力机制,提升语义分割的实时效率与精度。

ContextFormer: Redefining Efficiency in Semantic Segmentation

  • 在瓶颈层结合CNN与ViT优势,动态建模多尺度特征。
  • 在多个数据集上达到最新最高mIoU,实现实时推理性能突破。
  • 适合追求高精度与低延迟的视觉应用开发者参考。

语义分割为图像像素分配标签,是计算机视觉中的关键任务但极具挑战。传统卷积方法虽擅长捕捉局部依赖,却难以处理长程关系;视觉变换器(ViTs)虽能有效建模全局上下文,但计算开销大,尤其对高分辨率输入。现有研究多聚焦编码器优化,而忽略了瓶颈层这一关键瓶颈——正是提升性能与效率的核心区域。本文提出ContextFormer,一种混合框架,通过在瓶颈层融合CNN与ViT优势,实现效率、准确率与鲁棒性的平衡,适用于实时语义分割。其高效性由三个协同模块驱动:分层多尺度表征的令牌金字塔提取模块(TPEM),支持动态尺度感知特征建模的Transformer与分支深度卷积块(Trans-BDC),以及增强空间与上下文一致性的特征融合模块(FMM)。在ADE20K、Pascal Context、CityScapes和COCO-Stuff数据集上的大量实验表明,ContextFormer显著优于现有模型,达成最新的mIoU成绩,树立了效率与性能的新基准。代码将在论文接受后公开。

原文摘要 · Abstract (English)

Semantic segmentation assigns labels to pixels in images, a critical yet challenging task in computer vision. Convolutional methods, although capturing local dependencies well, struggle with long-range relationships. Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands, especially for high-resolution inputs. Most research optimizes the encoder architecture, leaving the bottleneck underexplored - a key area for enhancing performance and efficiency. We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation. The framework's efficiency is driven by three synergistic modules: the Token Pyramid Extraction Module (TPEM) for hierarchical multi-scale representation, the Transformer and Branched DepthwiseConv (Trans-BDC) block for dynamic scale-aware feature modeling, and the Feature Merging Module (FMM) for robust integration with enhanced spatial and contextual consistency. Extensive experiments on ADE20K, Pascal Context, CityScapes, and COCO-Stuff datasets show ContextFormer significantly outperforms existing models, achieving state-of-the-art mIoU scores, setting a new benchmark for efficiency and performance. The codes will be made publicly available upon acceptance.

语义分割混合架构实时推理ViT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。