arXiv:2606.02569cs.CVcs.AI2026-06被引 2

用预测性视觉码减少视频冗余,提升多模态大模型效率

AdaCodec: A Predictive Visual Code for Video MLLMs

论文配图:AdaCodec: A Predictive Visual Code for Video MLLMs
图 1 · 摘自论文原文
  • 根据场景可预测性动态决定传参考帧或变化编码
  • 32k token仅需1/7预算,长视频性能超基准模型
  • 显著缩短首帧响应时间,适合实时视频应用

视频具有时间冗余:相邻帧通常共享大部分物体、背景和布局。现有视频多模态大模型(video MLLMs)通常将每帧独立编码为RGB图像,导致视觉标记重复已有内容。我们提出更直接的视频接口:仅当先前上下文无法良好预测时才发送完整参考帧,否则传输帧间变化的紧凑描述。我们称此为“预测性视觉码”,并将其应用于video MLLMs,构建出AdaCodec。当条件预测成本高时,AdaCodec才使用全部视觉标记传参考帧;否则,以紧凑的P-token编码运动与预测残差。在全部11个基准测试中,AdaCodec在匹配视觉标记预算下优于Qwen3-VL-8B的逐帧RGB基线。即使仅用1/7预算(32k tokens),AdaCodec在所有长视频任务上仍超越224k基线;在5个通用视频任务中,平均得分提升,并将首次响应时间从9.26秒大幅压缩至1.62秒。

原文摘要 · Abstract (English)

Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames. This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a \emph{predictive visual code}, and instantiate it for video MLLMs as \textbf{AdaCodec}. AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens. Across all eleven benchmarks, AdaCodec improves over the Qwen3-VL-8B per-frame RGB baseline at a matched visual-token budget. Even at $1/7$ the budget, AdaCodec with 32k tokens surpasses the 224k baseline on all long-video benchmarks; on five general-video benchmarks, it raises the average score while substantially cutting time-to-first-token from 9.26s to 1.62s.

视频生成多模态高效编码大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。