压缩视频模型时保留语义,用双路教学提升精度。
Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

- 冻结主干网络,仅优化LoRA适配器,结合闪速视觉压缩。
- 在10%保留率下保持原模型97%准确率,匹配100%相对性能。
- 适合需要高效推理的视频理解场景,尤其对资源受限设备友好。
适应视频视觉语言模型(VLMs)计算成本高,因视频输入产生大量视觉标记,导致微调和推理开销大。尽管视觉标记压缩可降低开销,但直接在压缩输入上适应常引发语义漂移与性能显著下降。本文提出基于固定标记预算的参数高效微调框架Token-Budget Distillation(TBD)。TBD冻结预训练主干,仅更新LoRA适配器,并在视频路径中集成FlashVID视觉标记压缩。为在压缩下保持全标记语义,TBD采用双路教师-学生架构:全标记教师提供稳定监督,压缩学生通过任务损失、答案区域KL散度、真实标注锚定的边界散度及可靠性感知的知识蒸馏控制进行优化。该设计使学生模型在极端标记压缩下仍能恢复全标记模型的语义行为。我们在三个视频VLM主干(LLaVA-Video、LLaVA-OneVision、Qwen3-VL-8B-Instruct)上评估TBD,覆盖四个视频理解基准。在中等与激进压缩下,TBD始终优于仅压缩基线。在保留率R=10%时,LLaVA-Video保持97.0%平均准确率;在相同条件下,LLaVA-OneVision取得58.4平均分,达到100.0%相对准确率。
原文摘要 · Abstract (English)
Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。