提出ARM模块，让CLIP在开放词汇语义分割中更准更快
ARM: A Learnable, Plug-and-Play Module for CLIP-based Open-vocabulary Semantic Segmentation
- 用可学习的注意力模块动态融合深层与浅层特征
- 在多个基准上提升性能，推理开销几乎为零
- 训练一次即可通用适配多种无需训练的分割框架
开放词汇语义分割（OVSS）受限于CLIP粗粒度的图像级表征，缺乏像素级细节。现有免训练方法要么依赖昂贵的外部基础模型（如SAM、DINO），要么使用静态手工规则处理CLIP内部特征，前者计算成本高，后者效果不佳。本文提出注意力精炼模块（ARM），一个轻量级可学习模块，有效挖掘并优化CLIP内在潜力。不同于静态融合方法，ARM通过语义引导的跨注意力块，利用鲁棒的深层特征（K, V）选择并精炼细节丰富的浅层特征（Q），再经自注意力块增强。核心创新在于“训练一次，随处可用”范式：在通用数据集（如COCO-Stuff）上训练一次后，可作为通用即插即用后处理器，适配多种免训练框架。大量实验表明，ARM在多个基准上持续提升基线性能，且推理开销极低，建立了一种高效可靠的免训练OVSS新范式。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) is fundamentally hampered by the coarse, image-level representations of CLIP, which lack precise pixel-level details. Existing training-free methods attempt to resolve this by either importing priors from costly external foundation models (e.g., SAM, DINO) or by applying static, hand-crafted heuristics to CLIP's internal features. These approaches are either computationally expensive or sub-optimal. We propose the Attention Refinement Module (ARM), a lightweight, learnable module that effectively unlocks and refines CLIP's internal potential. Unlike static-fusion methods, ARM learns to adaptively fuse hierarchical features. It employs a semantically-guided cross-attention block, using robust deep features (K, V) to select and refine detail-rich shallow features (Q), followed by a self-attention block. The key innovation lies in a ``train once, use anywhere" paradigm. Trained once on a general-purpose dataset (e.g., COCO-Stuff), ARM acts as a universal plug-and-play post-processor for diverse training-free frameworks. Extensive experiments show that ARM consistently boosts baseline performance on multiple benchmarks with negligible inference overhead, establishing an efficient and effective paradigm for training-free OVSS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。