不训练的双分支模型提升开放词汇分割精度
DouC: Dual-Branch CLIP for Training-Free Open-Vocabulary Segmentation
- 双分支设计分别优化局部可靠性和空间结构一致性
- 在8个基准上均超越现有免训练方法，且无额外参数
- 适合需要零样本泛化能力的视觉分割场景
开放词汇语义分割需在支持任意类别集合的前提下进行像素级标注。基于CLIP的免训练方法虽具备强零样本泛化能力，但通常依赖单一推理机制，难以同时处理不可靠局部标记和空间连贯性不足的问题。本文提出DouC，一种免训练的双分支CLIP框架，将密集预测分解为两个互补组件：OG-CLIP通过轻量级推理时标记门控提升块级可靠性；FADE-CLIP则借助冻结视觉基础模型引导的代理注意力注入外部结构先验。两分支在logit层融合，使局部标记可靠性与结构感知块交互共同影响最终预测，可选实例级校正作为后处理。DouC不引入任何可学习参数，无需重训练，保持CLIP的零样本泛化能力。在八个基准及多个CLIP骨干网络上的大量实验表明，DouC持续优于先前免训练方法，且随模型容量提升表现出良好扩展性。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation requires assigning pixel-level semantic labels while supporting an open and unrestricted set of categories. Training-free CLIP-based approaches preserve strong zero-shot generalization but typically rely on a single inference mechanism, limiting their ability to jointly address unreliable local tokens and insufficient spatial coherence. We propose DouC, a training-free dual-branch CLIP framework that decomposes dense prediction into two complementary components. OG-CLIP improves patch-level reliability via lightweight, inference-time token gating, while FADE-CLIP injects external structural priors through proxy attention guided by frozen vision foundation models. The two branches are fused at the logit level, enabling local token reliability and structure-aware patch interactions to jointly influence final predictions, with optional instance-aware correction applied as post-processing. DouC introduces no additional learnable parameters, requires no retraining, and preserves CLIP's zero-shot generalization. Extensive experiments across eight benchmarks and multiple CLIP backbones demonstrate that DouC consistently outperforms prior training-free methods and scales favorably with model capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。