arXiv:2605.06592cs.CVcs.AI2026-05中稿 · ance

用高阶排序和教师注入,让图文预训练更懂局部细节。

DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency

论文配图:DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
图 1 · 摘自论文原文
  • 用DINOv3教师指导学生模型,融合多尺度特征并保持跨模态对齐。
  • 引入高阶排序损失,最优排序阶数R*=3,显著提升细粒度识别能力。
  • 仅用1个8卡H100节点72小时训练,适合追求精度的视觉-语言研究者。

对比式图文预训练(CLIP)存在两大结构性缺陷:对称InfoNCE损失忽略批次内未匹配对的相对顺序,全局池化将视觉表征压缩为语义瓶颈,难以捕捉细粒度局部结构。RANKCLIP通过列表级Plackett-Luce排序一致性损失部分解决第一问题,但模型仅为一阶且仍受第二问题影响。本文提出DINORANKCLIP,联合解决二者。核心是将冻结的DINOv3教师注入对比主干,通过双分支轻量学生与多尺度融合模块(含通道-空间注意力、自注意力精炼器及冲突感知门),保留跨模态对齐至一阶。同时引入高阶Plackett-Luce排序模型,每个位置效用增益包含注意力参数化的成对与三元组转移项;该族系包含CLIP(零阶)和RANKCLIP(一阶)作为特例,所有基准上最优阶数均为R*=3。全量实证研究——阶数扫描、五数据集细粒度探针、四节点模态鸿沟分析、六变体融合消融——在单个八卡H100节点上72小时内完成,训练全程基于Conceptual Captions 3M。DINORANKCLIP在相同计算量下持续优于CLIP、CyCLIP、ALIP和RANKCLIP,尤其在细粒度与分布外评估中表现最佳,最能体现局部结构推理能力。

原文摘要 · Abstract (English)

Contrastive language-image pretraining (CLIP) suffers from two structural weaknesses: the symmetric InfoNCE loss discards the relative ordering among unmatched in-batch pairs, and global pooling collapses the visual representation into a semantic bottleneck that is poorly sensitive to fine-grained local structure. RANKCLIP partially addresses the first issue with a list-wise Plackett-Luce ranking-consistency loss, but its model is strictly first-order and inherits the second weakness untouched. We propose DINORANKCLIP, a pretraining framework that addresses both jointly. Our principal contribution is injecting a frozen DINOv3 teacher into the contrastive trunk through a dual-branch lightweight student and a multi-scale fusion module with channel-spatial attention, a self-attention refiner, and a conflict-aware gate that preserves the cross-modal alignment up to first order. Complementarily, we introduce a high-order Plackett-Luce ranking model in which the per-position utility is augmented with attention-parameterised pairwise and tuple-wise transition terms; the family contains CLIP and RANKCLIP as nested zero-order and first-order special cases, and the optimal order on every benchmark is $R^*=3$. The full empirical study -- order sweep, Fine-grained Probe on five datasets, four-node Modality-Gap analysis, six-variant Fusion ablation -- fits in 72 hours on a single eight-GPU H100 node and trains entirely on Conceptual Captions 3M. DINORANKCLIP consistently outperforms CLIP, CyCLIP, ALIP, and RANKCLIP under matched compute, with the largest relative gains on the fine-grained and out-of-distribution evaluations that most directly stress local structural reasoning.

图文预训练高阶排序知识蒸馏DINOv3

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。