用文本指导图像特征保留，提升视觉语言对齐效果
TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
- 通过稀疏自编码器分离图像特征，按文本内容选择性重建
- 在合成和真实数据上均提升跨模态检索性能，长文本提升更明显
- 适合需要精准图文对齐的应用，如图像搜索与生成
视觉语言模型（如CLIP）因其共享的图像-文本嵌入空间而广泛用于各类任务。然而，图像与文本嵌入常存在对齐不足的问题，影响下游性能。研究表明，这源于信息失衡：图像包含的信息多于其描述性标题所传达的内容。本文提出TEVI框架，利用标题作为信号，决定从图像嵌入中保留哪些内容。具体地，采用稀疏自编码器分解图像嵌入，并训练掩码模块根据给定标题选择性重建嵌入。在合成标题的受控实验中，TEVI能有效保留标题描述的属性，同时丢弃无关信息。将TEVI应用于自然图像训练的CLIP模型后，在粗粒度短标题（MS COCO、Flickr）和细粒度长标题（IIW、DOCCI）基准上均实现检索性能提升，尤其在信息丰富的标题上增益更强，并在RoCOCO鲁棒性测试中表现更优。
原文摘要 · Abstract (English)
Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space. Despite this, the image and text embeddings are often poorly aligned, affecting downstream performance. Recent work has shown that this can be attributed to an information imbalance: images contain more information than their captions describe. In this work, we propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings. Specifically, we use sparse autoencoders to disentangle image embeddings and train a masking module to selectively reconstruct the embedding based on a given caption. In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others. By applying TEVI to CLIP models trained on natural images, we further achieve improved retrieval performance across coarse-grained short-caption (MS COCO, Flickr) and fine-grained long-caption (IIW, DOCCI) benchmarks, with stronger gains on richer captions, and improved robustness on the RoCOCO benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。