arXiv:2512.20157cs.CV2025-12被引 1

用多教师蒸馏训练高效视觉基础模型,提升数据与计算效率。

SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models

  • 设计不对称关系蒸馏损失,保留教师模型几何特性。
  • 采用令牌平衡批处理,跨分辨率稳定学习且不降性能。
  • 用分层聚类采样提升样本效率,适合追求高性能的开发者。

通过多教师蒸馏训练视觉基础模型可实现统一视觉表征,但其学习动态与数据效率仍待深入研究。本文系统探讨了该方法,并提出SigLino——一种高效的凝聚型视觉基础模型家族,同时从SigLIP2和DINOv3蒸馏知识至密集型与专家混合(MoE)学生模型。实验表明:(1)提出的不对称关系-知识蒸馏损失可保持各教师的几何特性并实现有效知识迁移;(2)令牌平衡批处理将不同分辨率图像打包进统一令牌预算序列,稳定了跨分辨率表示学习且未牺牲性能;(3)引入分层聚类与采样策略,显著提升多教师蒸馏的样本效率,优于随机采样;(4)所获表征在早期融合的接地视觉语言模型(Grounding-VLM)上表现优异,超越从零训练模型。基于上述成果,我们构建了包含200M图像的OpenLVD200M数据集。以专家混合架构实例化的SigLino-MoE,可替代传统ViT→LLM堆叠结构,初始化早期融合的Grounding-VLM并实现更优性能。论文发布OpenLVD200M及五个蒸馏检查点(含MoE与密集变体)。

原文摘要 · Abstract (English)

Vision foundation models trained via multi-teacher distillation offer a promising path toward unified visual representations, yet the learning dynamics and data efficiency of such approaches remain underexplored. In this paper, we systematically study multi-teacher distillation for vision foundation models and identify key factors that enable training at lower computational cost. We introduce SigLino, an efficient family of agglomerative vision foundation models that distill knowledge from SigLIP2 and DINOv3 simultaneously into Dense and Mixture-of-Experts students. We show that (1) our Asymmetric Relation-Knowledge Distillation loss preserves the geometric properties of each teacher while enabling effective knowledge transfer, (2) token-balanced batching that packs varying-resolution images into sequences with uniform token budgets stabilizes representation learning across resolutions without sacrificing performance, (3) hierarchical clustering and sampling of training data, typically reserved for self-supervised learning, substantially improves sample efficiency over random sampling for multi-teacher distillation, and (4) the resulting representations transfer effectively to early-fusion Grounding-VLMs, outperforming models trained from scratch. By combining these findings, we curate OpenLVD200M, a 200M-image corpus that demonstrates superior efficiency for multi-teacher distillation. Instantiated in a Mixture-of-Experts, our SigLino-MoE initializes an early-fusion Grounding-VLM that replaces the conventional ViT->LLM stack, demonstrating improved performance compared to a model trained from scratch. We release OpenLVD200M and five distilled checkpoints comprising MoE and dense variants.

视觉模型知识蒸馏多教师高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。