无需训练即可实现高质量开放词汇分割,靠的是统一的区域抽象框架。
SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

- 以区域为中心构建交互图,统一处理特征融合与空间支持。
- 在8个基准上提升结构质量与上下文推理能力,显著改善实例对齐效果。
- 适合追求零训练开集分割、注重推理效率的研究者或应用开发者。
训练自由的开放词汇分割受限于缺乏推理抽象。冻结的视觉语言特征在图像块级别生成,但稠密预测需要一个能同时处理特征交互、空间支持、上下文恢复和基于检索的修正的单元。本文提出SCI-CLIP,一种以区域为中心的推理框架,核心思想是同一区域抽象应贯穿稠密开放词汇预测的全过程。SCI-CLIP首先在冻结视觉标记上构建区域一致性交互图,随后通过该图传播值重建稠密特征,并仅在局部证据不足时引入选择性跨窗支持。相同的区域抽象用于构建和查询离线参考记忆,使示例检索与预测单元对齐。SCI-CLIP将冻结的CLIP型特征转化为空间连贯、上下文感知且可检索的稠密预测,全程无需训练。该方法在8个基准上持续提升预测结构质量、上下文推理鲁棒性及示例修正对齐度,显著增强开放词汇分割性能。项目代码已公开:https://github.com/mzamini92/SCICLIP。
原文摘要 · Abstract (English)
Training-free open-vocabulary segmentation remains limited by a missing inference abstraction. Frozen vision-language features are produced at patch level, yet dense prediction requires a unit that simultaneously governs feature interaction, spatial support, contextual recovery, and retrieval-based correction. We present SCI-CLIP, a segment-centric inference framework built around the principle that the same region abstraction should organize all stages of dense open-vocabulary prediction. SCI-CLIP first induces a region-consistent interaction graph over frozen visual tokens, then reconstructs dense features by propagating values over this graph, augmenting them with selective cross-window support only where local evidence is insufficient. The same segment abstraction is subsequently used to construct and query an offline reference memory, aligning exemplar retrieval with the units on which prediction is made. SCI-CLIP turns frozen CLIP-style features into spatially coherent, context-aware, and retrieval-compatible dense predictions without any training. SCI-CLIP consistently improves the structural quality of dense predictions, the robustness of contextual reasoning, and the alignment of exemplar-based correction, yielding stronger open-vocabulary segmentation across eight benchmarks. Project code is available at: https://github.com/mzamini92/SCICLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。