arXiv:2508.11256cs.CVcs.AI2025-08被引 8

让视觉模型看清开放词汇下的物体细节,提升定位与分割精度。

Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception

  • 分离图像特征中的内容与上下文信息,分别优化局部辨识与空间一致性。
  • 在多个任务上达到最新水平,2D/3D检测与分割、视频实例分割均领先。
  • 适合研究开放词汇视觉感知、模型泛化能力的学者与工程师。

密集视觉感知任务受限于预定义类别,难以应对现实场景中无限多样的视觉概念。尽管像CLIP这样的视觉-语言模型在开放词汇任务中展现出潜力,但其直接应用于密集感知时表现不佳,主要源于局部特征表示不足。本文观察到,CLIP的图像标记难以有效聚合空间或语义相关区域的信息,导致特征缺乏局部判别力和空间一致性。为此,我们提出DeCLIP框架,通过解耦自注意力模块,分别获取“内容”与“上下文”特征。上下文特征通过联合蒸馏视觉基础模型(VFMs)的语义关联和扩散模型的对象完整性线索,增强空间一致性;内容特征则通过与图像裁片表示对齐,并受VFMs区域关联约束,提升局部判别力。大量实验表明,DeCLIP为开放词汇密集感知奠定了坚实基础,在2D检测与分割、3D实例分割、视频实例分割及6D物体位姿估计等任务中持续取得最先进性能。

原文摘要 · Abstract (English)

Dense visual perception tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense perception often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain ``content'' and ``context'' features respectively. \revise{The context features are enhanced by jointly distilling semantic correlations from Vision Foundation Models (VFMs) and object integrity cues from diffusion models, thereby enhancing spatial consistency. In parallel, the content features are aligned with image crop representations and constrained by region correlations from VFMs to improve local discriminability. Extensive experiments demonstrate that DeCLIP establishes a solid foundation for open-vocabulary dense perception, consistently achieving state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.} Code is available at https://github.com/xiaomoguhz/DeCLIP

开放词汇密集感知视觉语言模型特征解耦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。