让AI理解长图文时，先找局部再组合整体，提升细节识别能力。
Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding
- 分步学习：先对齐局部语义与图像区域，再融合成全局表示。
- 在6个长文本检索任务上达到顶尖效果，3000万图文对训练。
- 无需区域标注，自动定位文字描述对应的图像位置，适合细粒度理解场景。
视觉语言模型如CLIP在理解长而细节丰富的文本时，常依赖主要场景线索，忽略细微视觉证据。本文提出一种层次化视觉-语言学习范式，将场景视为部分到整体的组合：在形成整体表征前，应先识别图像中出现的语义部分及其位置。为此，提出CAFT（Cross-domain Alignment of Forests and Trees）模型，联合学习中间表示中的局部文图对齐与最终表示中的全局图像-文本对齐。利用长文本的组织结构——局部描述常对应场景部分——CAFT采用细粒度到粗粒度的图像编码器与部分-整体文本编码器，逐步发现局部语义并组合为全局表征。在3000万图像-文本对上训练后，CAFT在六个长文本检索基准上达到当前最优性能，并表现出强劲的扩展能力。实验表明，该模型能学习到细粒度表征，在无显式区域监督下实现文本语义在图像区域上的准确定位。
原文摘要 · Abstract (English)
Vision-language models such as CLIP often struggle to faithfully understand long, detail-rich captions, relying on dominant scene cues while overlooking fine-grained visual evidence. We propose a hierarchical vision-language learning principle for understanding scenes as part-to-whole compositions: before forming a whole-scene representation, a model should uncover what semantic parts appear where in the image. To this end, we propose CAFT (Cross-domain Alignment of Forests and Trees), a vision-language model that jointly learns local text-region alignment at intermediate representations and global image-text alignment at the final representation. Exploiting the organization of long captions, where local descriptions often correspond to scene parts, CAFT employs a fine-to-coarse image encoder and a part-whole text encoder to discover localized part semantics and progressively compose them into a global image-text representation. Trained on 30M image-text pairs, CAFT achieves state-of-the-art performance on six long-text retrieval benchmarks and exhibits strong scaling behavior. Experiments show that CAFT learns fine-grained representations that localize textual semantics in image regions without explicit region-level supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。