arXiv:2602.20089cs.CVcs.AI2026-02中稿 · CVPR被引 1

用边缘图增强视觉语言模型,让图文对齐更精准。

StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues

  • 提取图像边缘图作为结构特征,引导文本聚焦结构描述。
  • 在跨模态检索中优于现有方法,尤其在细节丰富的长文本上提升显著。
  • 可插即用,适合需要提升图文对齐精度的研究与应用。

边缘表示是视觉理解的基础线索,源于早期视觉研究并持续重要。本文将其扩展至视觉-语言对齐任务,证明分离并对齐跨模态的结构线索能显著提升细粒度长文本(如详细描述性标题)上的微调效果,重点优化跨模态检索性能。提出 StructXLIP,一种微调对齐范式:通过 Canny 等方法提取边缘图,作为图像视觉结构的代理;同时筛选对应文本,强调结构相关描述,使文本变为“结构中心”。微调过程中,在标准对齐损失外引入三项结构中心损失:(i) 边缘图与结构化文本对齐;(ii) 局部边缘区域与文本片段匹配;(iii) 边缘图与彩色图像连接,防止表示漂移。理论上,标准 CLIP 最大化视觉与文本嵌入间的互信息,而 StructXLIP 还额外最大化多模态结构表示间的互信息。该辅助优化更具挑战性,引导模型趋向更鲁棒、语义稳定的极小值点,增强视觉-语言对齐。实验表明,其在通用与专用领域跨模态检索中均超越当前最优方法,且可作为通用增强方案无缝集成于未来框架中。代码与预训练模型已公开于 https://github.com/intelligolabs/StructXLIP。

原文摘要 · Abstract (English)

Edge-based representations are fundamental cues for visual understanding, a principle rooted in early vision research and still central today. We extend this principle to vision-language alignment, showing that isolating and aligning structural cues across modalities can greatly benefit fine-tuning on long, detail-rich captions, with a specific focus on improving cross-modal retrieval. We introduce StructXLIP, a fine-tuning alignment paradigm that extracts edge maps (e.g., Canny), treating them as proxies for the visual structure of an image, and filters the corresponding captions to emphasize structural cues, making them "structure-centric". Fine-tuning augments the standard alignment loss with three structure-centric losses: (i) aligning edge maps with structural text, (ii) matching local edge regions to textual chunks, and (iii) connecting edge maps to color images to prevent representation drift. From a theoretical standpoint, while standard CLIP maximizes the mutual information between visual and textual embeddings, StructXLIP additionally maximizes the mutual information between multimodal structural representations. This auxiliary optimization is intrinsically harder, guiding the model toward more robust and semantically stable minima, enhancing vision-language alignment. Beyond outperforming current competitors on cross-modal retrieval in both general and specialized domains, our method serves as a general boosting recipe that can be integrated into future approaches in a plug-and-play manner. Code and pretrained models are publicly available at: https://github.com/intelligolabs/StructXLIP.

视觉语言边缘感知跨模态对齐CLIP改进

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。