通过视频编码思路压缩视觉令牌，提升视频多模态大模型效率
Visual Token Coding for Video Multimodal Large Language Models
- 基于视频帧预测与残差分析，结构化压缩视觉令牌
- 令牌预算减半时仍保持100.1%性能，25%预算下保留97.8%
- 无需微调模型，可直接部署于Qwen3-VL等多模型
本文提出一种面向视频多模态大模型（MLLM）的新令牌压缩范式——视觉令牌编码（VTC）。受经典视频编码技术（如HEVC）启发，VTC通过预测视频的I/P帧并测量逐帧残差，评估令牌冗余性以实现结构化压缩。在此基础上，我们引入动态设计：动态分辨率输入（DyRSO）、动态令牌分配（DyTA）和空间覆盖Top-K（SC-TopK），形成增强版方法$VTC_{Dy}$。我们将VTC应用于三种MLLM，在多个视频理解基准上验证其有效性。实验结果表明，当令牌预算降低50%时，VTC$_{ ext{Dy}}$在Qwen3-VL上平均性能保留率达100.1%；预算降至25%时仍保持97.8%的平均性能。作为即插即用设计，VTC无需对MLLM进行额外调优。代码已开源：https://github.com/Msr233/VTC。
原文摘要 · Abstract (English)
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach $VTC_{Dy}$. To validate VTC, we apply it to three MLLMs and conduct experiments on multiple video understanding benchmarks. The experimental results show that VTC$_{\mathrm{Dy}}$ achieves an average performance retention of 100.1% with a 50% token budget for Qwen3-VL, while still retaining 97.8% of the average performance when the token budget is reduced to 25%. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is available at https://github.com/Msr233/VTC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。