让机器理解图像间的内在关系,而非仅看表面相似
Relational Visual Similarity
- 基于关系逻辑构建新数据集,捕捉图像间深层结构对应
- 在11.4万张图像上训练模型,显著提升关系相似性识别能力
- 适合研究认知视觉、跨模态理解的学者使用
人类不仅感知属性相似性,还能识别关系相似性:苹果与桃子都是红润水果,地球与桃子也存在类似结构——地壳、地幔、地核分别对应桃皮、果肉、果核。这种关系感知能力被认知科学家视为人类独有的特征。然而,当前主流视觉相似度度量方法(如LPIPS、CLIP、DINO)仅关注外观属性,无法捕捉人类所感知的丰富关系相似性。为此,本文将关系图像相似性定义为可度量问题:当两图内部元素间的关系或功能对应时,即使外观不同,也视为关系相似。我们构建了一个包含11.4万张图像-描述对的数据集,其中描述匿名化,仅表达场景的底层关系逻辑。在此数据集上微调视觉-语言模型,实现对图像关系相似性的测量。该模型首次实现了按深层关系结构而非表观特征连接图像,揭示了现有模型在关系理解上的重大缺陷,为视觉计算开辟新方向。
原文摘要 · Abstract (English)
Humans do not just see attribute similarity -- we also see relational similarity. An apple is like a peach because both are reddish fruit, but the Earth is also like a peach: its crust, mantle, and core correspond to the peach's skin, flesh, and pit. This ability to perceive and recognize relational similarity, is arguable by cognitive scientist to be what distinguishes humans from other species. Yet, all widely used visual similarity metrics today (e.g., LPIPS, CLIP, DINO) focus solely on perceptual attribute similarity and fail to capture the rich, often surprising relational similarities that humans perceive. How can we go beyond the visible content of an image to capture its relational properties? How can we bring images with the same relational logic closer together in representation space? To answer these questions, we first formulate relational image similarity as a measurable problem: two images are relationally similar when their internal relations or functions among visual elements correspond, even if their visual attributes differ. We then curate 114k image-caption dataset in which the captions are anonymized -- describing the underlying relational logic of the scene rather than its surface content. Using this dataset, we finetune a Vision-Language model to measure the relational similarity between images. This model serves as the first step toward connecting images by their underlying relational structure rather than their visible appearance. Our study shows that while relational similarity has a lot of real-world applications, existing image similarity models fail to capture it -- revealing a critical gap in visual computing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。