构建图像对比数据集TWIN，提升视觉语言模型的细粒度感知能力。
Same or Not? Enhancing Visual Perception in Vision-Language Models
- 设计TWIN数据集，通过对比相似图像判断是否为同一物体，强化模型对细微差异的敏感度。
- 在未见领域如艺术、动植物上，模型细粒度识别准确率最高提升19.3%。
- 适合关注视觉细节、跨域泛化能力的研究者和开发者使用。
视觉语言模型（VLMs）虽在广泛视觉理解上表现优异，但仍存在粗粒度、视觉偏见及忽略细微视觉特征的问题。现有训练语料多强调通用识别（如‘是猫还是狗？’），忽视细粒度感知。为此，本文提出TWIN：一个包含561,000组图像对的大型数据集，任务为判断两幅视觉相似图像是否描述同一对象，引导模型关注细微视觉线索。该数据集涵盖多种日常物体，在不同场景、视角与外观下具有多样性。在TWIN上微调的VLMs在未见领域（如艺术、动物、植物、地标）的细粒度识别中取得显著提升。为量化效果，我们引入FGVQA基准套件，包含12,000个查询，复用多个领域的细粒度识别与检索数据集。现有VLMs在FGVQA上表现不佳，但在TWIN上微调后性能最高提升19.3%，且不影响通用VQA任务表现。分析表明，对象标注规模与性能呈正相关，规模是关键因素。我们建议将TWIN作为开源VLM训练语料的即插即用补充，推动未来模型的感知精度提升。
原文摘要 · Abstract (English)
Vision-language models (VLMs) excel at broad visual understanding but remain coarse-grained, exhibit visual biases, and miss subtle visual details. Existing training corpora reinforce this limitation by emphasizing general recognition ("Is it a cat or a dog?") over fine-grained perception. To address this, we introduce a new training corpus and task designed to enhance the perceptual abilities of VLMs. TWIN is a large-scale dataset of 561,000 image-pair queries that task models to determine whether two visually similar images depict the same object, encouraging attention to nuanced visual cues. The dataset spans a diverse range of everyday objects across contexts, viewpoints, and appearances. Fine-tuning VLMs on TWIN yields notable gains in fine-grained recognition, even on unseen domains such as art, animals, plants, and landmarks. To quantify these gains, we introduce FGVQA, a benchmark suite of 12,000 queries that repurposes fine-grained recognition and retrieval datasets from multiple domains. While existing VLMs struggle on FGVQA, when fine-tuned on TWIN they improve by up to 19.3%, without compromising performance on general VQA benchmarks. Finally, our TWIN dataset scales favorably with object annotations, and our analysis shows that scale is key to performance. We envision TWIN as a drop-in addition to open-source VLM training corpora, advancing perceptual precision of future models. Project webpage: https://glab-caltech.github.io/twin/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。