TDDN提升图像谜题理解，通过文本对齐增强细粒度视觉感知。
TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding
- 融合DINOv3与CleanDIFT特征，构建文本对齐的感知编码器
- 在细粒度分割任务上超越CLIP三倍以上准确率（如ADE20K达18.11 mIoU）
- 适合需要高精度空间理解的视觉推理场景，如图像谜题解析
结构化视觉推理（如图像拼图）需要精细的视觉感知能力，而现有基于CLIP的视觉语言模型因采用ViT主干网络，牺牲了细粒度细节以换取高层语义。本文表明该损失会向下游任务传导。为恢复这一能力，我们融合DINOv3与CleanDIFT表示，构建感知编码器DiffusedDINO，并与RoBERTa-L对齐，形成文本对齐模型TDDN。仅用约590K对齐样本且冻结主干网络，TDDN在图像-文本检索上达到与CLIP相当的性能，在四个设置中的三个表现更优。同时，其稠密预测准确率超过CLIP三倍以上（ADE20K：5.20→18.11 mIoU，COCO-Stuff：7.35→24.44），尽管后者训练数据量巨大。TDDN在通用对比编码器中领先于分割基准，包括SigLIP 2。此外，我们引入新数据集Puzzle Perception，用于评估细粒度空间理解，TDDN在此上的分割准确率是CLIP的两倍（11.04→22.51 mIoU）。
原文摘要 · Abstract (English)
Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only $\sim$590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20 $\to$ 18.11 mIoU, COCO-Stuff 7.35 $\to$ 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP$\,$2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 $\to$ 22.51 mIoU).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。