利用预训练模型库实现双向知识迁移，提升各类任务性能
Simplifying Knowledge Transfer in Pretrained Models
- 让模型自动扮演学生或教师角色，双向迁移知识
- 图像分类中ViT-B性能提升1.4%，视频显著优于现有方法
- 适用于多模型协作，尤其适合想快速优化的开发者
预训练模型在深度学习中广泛应用，但不同设计的模型表现出截然不同的泛化能力，各自掌握独特数据洞察。本文提出利用大型公开模型库作为知识来源，设计一种数据划分策略，使预训练模型自主选择作为学生（吸收知识）或教师（传授知识）。在多个任务上验证了该方法的有效性：图像分类中，通过与ViT-T双向知识迁移，使ViT-B性能提升约1.4%；语义分割中，跨架构和同架构间知识迁移均显著提升所有评估指标；视频显著性预测达到新最佳水平。进一步扩展至多模型间的知识传递，所有参与模型均获得显著性能提升。
原文摘要 · Abstract (English)
Pretrained models are ubiquitous in the current deep learning landscape, offering strong results on a broad range of tasks. Recent works have shown that models differing in various design choices exhibit categorically diverse generalization behavior, resulting in one model grasping distinct data-specific insights unavailable to the other. In this paper, we propose to leverage large publicly available model repositories as an auxiliary source of model improvements. We introduce a data partitioning strategy where pretrained models autonomously adopt either the role of a student, seeking knowledge, or that of a teacher, imparting knowledge. Experiments across various tasks demonstrate the effectiveness of our proposed approach. In image classification, we improved the performance of ViT-B by approximately 1.4% through bidirectional knowledge transfer with ViT-T. For semantic segmentation, our method boosted all evaluation metrics by enabling knowledge transfer both within and across backbone architectures. In video saliency prediction, our approach achieved a new state-of-the-art. We further extend our approach to knowledge transfer between multiple models, leading to considerable performance improvements for all model participants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。