用大模型零训练生成按条件聚焦的图像嵌入，提升相似性任务表现。
Training-free Conditional Image Embedding Framework Leveraging Large Vision Language Models
- 通过提示大视觉语言模型用单个关键词描述图像，提取最后一层隐状态作为嵌入。
- 在多个条件相似性任务上优于无训练基线（如CLIP）和需训练的方法。
- 无需额外训练或领域先验，通用性强，适用于任意图像与条件组合。
条件图像嵌入是针对图像中特定方面（如颜色、风格）进行聚焦的特征表示，这一问题仍具挑战性。尽管近期视觉基础模型（如CLIP）提供丰富的图像表征，但其并非为指定条件设计。本文提出DIOR，一种利用大视觉语言模型（LVLM）生成条件图像嵌入的零训练方法。DIOR通过提示LVLM以与给定条件相关的单个词描述图像，提取其最后一层标记的隐藏状态向量作为条件图像嵌入。该方法无需额外训练或任务特定先验，可广泛应用于任意图像与条件。在条件图像相似性任务上的综合实验表明，DIOR超越现有零训练基线（包括CLIP），并在多种设置下优于需额外训练的方法。
原文摘要 · Abstract (English)
Conditional image embeddings are feature representations that focus on specific aspects of an image indicated by a given textual condition (e.g., color, genre), which has been a challenging problem. Although recent vision foundation models, such as CLIP, offer rich representations of images, they are not designed to focus on a specified condition. In this paper, we propose DIOR, a method that leverages a large vision-language model (LVLM) to generate conditional image embeddings. DIOR is a training-free approach that prompts the LVLM to describe an image with a single word related to a given condition. The hidden state vector of the LVLM's last token is then extracted as the conditional image embedding. DIOR provides a versatile solution that can be applied to any image and condition without additional training or task-specific priors. Comprehensive experimental results on conditional image similarity tasks demonstrate that DIOR outperforms existing training-free baselines, including CLIP. Furthermore, DIOR achieves superior performance compared to methods that require additional training across multiple settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。