通过局部结构对齐提升CLIP检索精度，解决形状混淆问题。
Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering
- 用匈牙利匹配重排局部邻域，强化结构一致性
- 查询条件引导局部方向调整，提升检索可控性
- 无需微调，在推理时即可改善组合式检索效果
CLIP检索通常被视为共享嵌入空间中的点对点相似性问题。尽管CLIP实现了强大的全局跨模态对齐，但许多检索失败源于局部几何不一致：相邻项排序错误，导致系统性混淆（如五边形与六边形）并产生分散、控制力弱的结果集。现有方法主要通过点对点相关性优化或微调来缓解此问题。本文将检索视为邻域对齐问题，提出：(1) 基于匈牙利匹配的邻域级重排，奖励结构一致性；(2) 查询条件化的局部引导，利用查询周围对比邻域的方向重塑检索结果。实验表明，这些方法在属性绑定和组合式检索任务上均取得性能提升。二者虽作用于局部邻域，但角色不同：重排强调对齐，局部引导则控制邻域结构。结果表明，检索质量与可控性高度依赖局部结构，且可在推理阶段利用，无需重新训练。
原文摘要 · Abstract (English)
CLIP retrieval is typically framed as a pointwise similarity problem in a shared embedding space. While CLIP achieves strong global cross-modal alignment, many retrieval failures arise from local geometric inconsistencies: nearby items are incorrectly ordered, leading to systematic confusions (e.g., pentagon vs. hexagon) and produces diffuse, weakly controlled result sets. Prior work largely optimizes for point wise relevance or finetuning to mitigate these problems. We instead view retrieval as a problem of neighborhood alignment. Our work introduces (1) neighborhood-level re-ranking via Hungarian matching, which rewards structural consistency; (2) query-conditioned local steering, where directions derived from contrastive neighborhoods around the query reshape retrieval. We show that these techniques improve retrieval performance on attribute-binding and compositional retrieval tasks. Together, these methods operate on local neighborhoods but serve different roles: re-ranking rewards alignment whereas local steering controls neighborhood structure. This shows that retrieval quality and controllability depend critically on local structure, which can be exploited at inference time without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。