用多采样教师提升视频理解模型训练稳定性
Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding
- 构建任务自适应教师池，通过多样本响应降低噪声影响
- 在多个视频理解数据集上实现显著性能提升，最高+3.6%
- 适合追求高鲁棒性视频理解模型的开发者使用
大型视觉语言模型（LVLM）的黑盒蒸馏通常依赖单个教师输出，导致多模态或时序场景下响应方差大、格式不一致。为解决这一问题，我们提出R-MSD（可靠多样本蒸馏）框架，显式建模教师采样方差以增强蒸馏稳定性。不同于单一教师响应，本方法利用任务自适应教师池，为封闭式与开放式推理提供鲁棒监督。通过融合质量感知信号匹配与对抗蒸馏目标，有效过滤教师噪声并最大化知识迁移。在多个视频理解基准上的实验证明，R-MSD持续优于单样本蒸馏方法。在相同训练预算下，我们还引入一个SFT+RL 4B基线，仅取得微弱提升；而我们的方法在4B学生模型上实现了显著改进：VideoMME提升1.5%，Video-MMMU提升3.2%，MathVerse提升3.6%。
原文摘要 · Abstract (English)
Traditional black-box distillation for Large Vision-Language Models (LVLMs) typically relies on a single teacher response per input, which often yields high-variance responses and format inconsistencies in multimodal or temporal scenarios. To mitigate this unreliable supervision, we propose R-MSD (Reliable Multi-Sample Distillation), a framework that explicitly models teacher sampling variance to enhance distillation stability. Rather than relying on a single teacher response, our approach leverages a task-adaptive teacher pool to provide robust supervision tailored to both closed-ended and open-ended reasoning. By integrating quality-aware signal matching with an adversarial distillation objective, our approach effectively filters teacher noise while maximizing knowledge transfer. Extensive evaluations across comprehensive video understanding benchmarks demonstrate that R-MSD consistently outperforms single sample distillation methods. We additionally include an original SFT+RL 4B baseline under the same training budget, which shows only marginal gains, while our method achieves significant improvements. With a 4B student model, our approach delivers gains on VideoMME (+1.5%), Video-MMMU (+3.2%), and MathVerse (+3.6%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。