轻量视频摘要模型通过分段推理与多阶段知识蒸馏，兼顾速度与准确率。
Early Exit and Multi Stage Knowledge Distillation in VLMs for Video Summarization
- 采用多阶段知识蒸馏和早期退出机制提升效率
- 在TVSum上达61.1的F1分数，仅用小模型
- 适合需要低延迟视频摘要的应用场景
我们提出DEEVISum（用于摘要的蒸馏早期退出视觉语言模型），一种面向片段式视频摘要的轻量、高效且可扩展的视觉语言模型。该模型融合文本与音频信号构建多模态提示，结合多阶段知识蒸馏（MSKD）与早期退出（EE），在性能与效率间取得平衡。相比基线蒸馏（0.5%），MSKD带来1.33%的绝对F1提升；EE使推理时间减少约21%，F1下降1.3点。在TVSum数据集上，最佳模型PaLI Gemma2 3B + MSKD实现61.1的F1分数，媲美更大模型，同时保持更低计算开销。代码与处理后的数据集已公开，支持后续研究。
原文摘要 · Abstract (English)
We introduce DEEVISum (Distilled Early Exit Vision language model for Summarization), a lightweight, efficient, and scalable vision language model designed for segment wise video summarization. Leveraging multi modal prompts that combine textual and audio derived signals, DEEVISum incorporates Multi Stage Knowledge Distillation (MSKD) and Early Exit (EE) to strike a balance between performance and efficiency. MSKD offers a 1.33% absolute F1 improvement over baseline distillation (0.5%), while EE reduces inference time by approximately 21% with a 1.3 point drop in F1. Evaluated on the TVSum dataset, our best model PaLI Gemma2 3B + MSKD achieves an F1 score of 61.1, competing the performance of significantly larger models, all while maintaining a lower computational footprint. We publicly release our code and processed dataset to support further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。