让学生模型根据教师不确定性自适应学习，提升多模态大模型性能。
Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
- 从贝叶斯视角建模师生关系，用吉布斯先验表示教师指导。
- 提出贝塔加权机制，自动调节学生对数据与教师的依赖程度。
- 在多模态VQA任务上优于现有方法，适合追求高鲁棒性的研究者。
知识蒸馏通过数据监督和教师指导实现学习，但如何平衡二者仍具挑战，因部分样本存在噪声，另一些则受教师不确定性影响。为此，本文提出贝塔加权知识蒸馏（Beta-KD），一种基于不确定性的自适应蒸馏框架，可动态调节学生模型对教师指导的依赖程度。具体而言，从统一的贝叶斯视角建模师生学习过程，将教师指导视为对学生激活的吉布斯先验，从而导出闭式、不确定性感知的加权机制，支持任意蒸馏目标及其组合。在多模态视觉问答（VQA）基准上的大量实验表明，从大型教师视觉语言模型（VLM）蒸馏学生模型能持续提升性能，且Beta-KD显著优于现有蒸馏方法。
原文摘要 · Abstract (English)
Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncertainty. This motivates the need for adaptively balancing data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student relies on teacher guidance. Specifically, we formulate teacher--student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and their combinations. Extensive experiments on multimodal VQA benchmarks demonstrate that distilling student Vision-Language Models from a large teacher VLM consistently improves performance. The results show that Beta-KD outperforms existing knowledge distillation methods. The code is available at https://github.com/Jingchensun/beta-kd.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。