arXiv:2503.16980cs.CVcs.AI2025-03NeurIPS被引 14

用离散令牌压缩视频,仅用原长0.07%的令牌保持高精度。

VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models

  • 通过自适应向量量化将连续视觉令牌转为紧凑离散代码本。
  • 在NextQA-MC上仅损失0.66%准确率,序列长度压缩至原长0.07%。
  • 适合资源受限环境下的高效视频大模型应用。

基于令牌的视频表示已成为使大语言模型理解视频内容的有前景方法。然而,现有压缩技术如剪枝和合并常破坏关键位置嵌入,并依赖于空间时间邻近像素采样的连续视觉令牌。仅移除少量令牌仍产生较长连续序列,难以满足视频大模型对计算效率与令牌数量平衡的极致压缩需求。本文提出极端短令牌压缩新任务,旨在用极少数离散令牌表示完整视频。我们提出VQToken框架:(i) 对ViT连续嵌入应用自适应向量量化以学习紧凑代码本;(ii) 通过令牌哈希函数保留时空位置,将每个网格级令牌分配给最近的代码本条目。在极端短令牌压缩任务中,VQToken将序列压缩至原长的0.07%,在NextQA-MC上仅损失0.66%准确率,同时在ActNet-QA、Long Video Bench和VideoMME上表现相当。我们还引入令牌信息密度(TokDense)指标,形式化固定长度与自适应长度子任务,两类设置均达当前最优。该方法显著降低理论复杂度,提升信息密度,大幅减少令牌数量,使资源受限环境下视频大模型成为可能。

原文摘要 · Abstract (English)

Token-based video representation has emerged as a promising approach for enabling large language models (LLMs) to interpret video content. However, existing token reduction techniques, such as pruning and merging, often disrupt essential positional embeddings and rely on continuous visual tokens sampled from nearby pixels with similar spatial-temporal locations. By removing only a small fraction of tokens, these methods still produce relatively lengthy continuous sequences, which falls short of the extreme compression required to balance computational efficiency and token count in video LLMs. In this paper, we introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens. We propose VQToken, a neural discrete token representation framework that (i) applies adaptive vector quantization to continuous ViT embeddings to learn a compact codebook and (ii) preserves spatial-temporal positions via a token hash function by assigning each grid-level token to its nearest codebook entry. On the Extreme Short Token Reduction task, our VQToken compresses sequences to just 0.07 percent of their original length while incurring only a 0.66 percent drop in accuracy on the NextQA-MC benchmark. It also achieves comparable performance on ActNet-QA, Long Video Bench, and VideoMME. We further introduce the Token Information Density (TokDense) metric and formalize fixed-length and adaptive-length subtasks, achieving state-of-the-art results in both settings. Our approach dramatically lowers theoretical complexity, increases information density, drastically reduces token counts, and enables efficient video LLMs in resource-constrained environments.

视频压缩离散表示大模型令牌优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。