让图像中每个物体都有专属嵌入，提升视觉与文本匹配精度
ObjEmbed: Towards Universal Multimodal Object Embeddings
- 将图像拆解为多个物体区域，生成语义与定位双嵌入
- 在18个基准上表现优异，物体匹配更准
- 适合需要精细定位的视觉理解任务
将物体与其对应文本描述对齐是视觉语言理解中的核心挑战和实际需求。尽管当前多模态嵌入模型在全局图像-文本对齐上表现良好，但在图像区域与特定短语之间的细粒度对齐上仍存在困难。本文提出ObjEmbed，一种新型多模态大语言模型嵌入框架，将输入图像分解为多个区域嵌入，每个对应一个独立物体，同时保留全局嵌入。该模型支持视觉定位、局部图像检索与全局图像检索等多种视觉理解任务。其具备三大特性：(1) 物体导向表示：通过为每个区域生成两个互补嵌入——物体嵌入用于语义匹配，IoU嵌入用于预测定位质量——最终匹配分数融合语义相似性与预测的交并比，实现更精准检索；(2) 多样性适应：可无缝处理区域级与图像级任务；(3) 高效编码：图像中所有物体及整图均在一次前向传播中完成编码，效率极高。在18个多样化的基准测试中表现出卓越的语义区分能力。
原文摘要 · Abstract (English)
Aligning objects with corresponding textual descriptions is a fundamental challenge and a realistic requirement in vision-language understanding. While recent multimodal embedding models excel at global image-text alignment, they often struggle with fine-grained alignment between image regions and specific phrases. In this work, we present ObjEmbed, a novel MLLM embedding model that decomposes the input image into multiple regional embeddings, each corresponding to an individual object, along with global embeddings. It supports a wide range of visual understanding tasks like visual grounding, local image retrieval, and global image retrieval. ObjEmbed enjoys three key properties: (1) Object-Oriented Representation: It captures both semantic and spatial aspects of objects by generating two complementary embeddings for each region: an object embedding for semantic matching and an IoU embedding that predicts localization quality. The final object matching score combines semantic similarity with the predicted IoU, enabling more accurate retrieval. (2) Versatility: It seamlessly handles both region-level and image-level tasks. (3) Efficient Encoding: All objects in an image, along with the full image, are encoded in a single forward pass for high efficiency. Superior performance on 18 diverse benchmarks demonstrates its strong semantic discrimination.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。