arXiv:2607.13805cs.CV2026-07

让图文匹配更合理,避免因描述角度不同导致的语义扭曲。

AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization

论文配图:AspectCLIP: Optimizing CLIP Representation Space via Aspect-Guided Consistency Regularization
图 1 · 摘自论文原文
  • 按文本相似性分组,只在相同描述角度内强化图文一致性。
  • 在视觉相似但描述角度不同的样本间放松约束,减少语义干扰。
  • 适用于需要精准图文对齐的场景,如跨模态检索与生成。

对比语言-图像预训练通过大规模对比学习构建共享表示空间。然而,现有方法施加全局一致性正则化时忽略了图像与文本间的固有信息不对称:图片描述通常仅聚焦于某一特定方面,导致视觉内容相似的图片可能配以完全不同的文字描述和语义信息。因此,全局正则化会无意中在描述不同方面的视觉相似图片之间施加约束,引入表示空间中的语义失真。我们提出AspectCLIP,一种重构一致性正则化的框架,以尊重这种一对多的结构特性。AspectCLIP首先基于文本相似性将训练样本划分为属性聚类,识别出语义一致的分组;然后在每个聚类内部实施全循环一致性,而跨聚类的正则化仅限于原型层面的比较。这种基于视角的正则化机制仅在图像与文本描述一致方面强制几何对齐,而在不同描述角度间保持灵活性。大量下游任务实验表明,AspectCLIP持续优于传统方法,并实现更结构化的表示空间。

原文摘要 · Abstract (English)

Contrastive Language-Image Pretraining learns a shared representation space through large-scale contrastive learning. However, existing methods that enforce global consistency regularization overlook a key challenge: the inherent information asymmetry between images and text: captions typically describe only one specific aspect of an image, thus images with similar visual content can be paired with completely divergent textual content and semantic information. Consequently, global regularizers inadvertently impose constraints between visually similar images whose captions describe divergent aspects, introducing semantic distortion into the representation space. We propose AspectCLIP, a framework that reformulates consistency regularization to respect this one-to-many structure. AspectCLIP first partitions training samples into attribute clusters based on textual similarity to identify aspect-coherent groups, then applies full cyclic consistency within each cluster while restricting cross-cluster regularization to prototype-level comparisons. This aspect-guided regularization enforces strict geometric alignment only when images and texts describe a consistent facet, while allowing flexibility across divergent aspects. Extensive experiments on downstream tasks demonstrate that AspectCLIP consistently outperforms traditional methods and achieves a more structured representation space.

对比学习图文对齐表示空间一致性正则

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。