根据信息量动态压缩视频，减少20%令牌且不损失性能。
InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression
- 基于信息论构建自适应压缩框架，按内容重要性分配令牌。
- 实现2.3倍压缩率，比现有方法少用20%令牌且性能相当。
- 适合需要高效视频建模的场景，如长视频理解与生成。
精确高效的离散视频标记化对长视频序列处理至关重要。然而，视频固有的复杂性和信息密度差异导致现有标记器在固定速率下压缩所有内容，造成冗余或信息丢失。受香农信息论启发，本文提出InfoTok，一种原理严谨的自适应视频标记化框架。我们严格证明，现有无数据依赖的训练方法在表示长度上非最优，并提出一种基于证据下界（ELBO）的算法，逼近理论最优。基于此框架，我们开发了一个Transformer-based自适应压缩器，实现动态标记化。实验表明，该方法达到业界领先压缩性能：节省20%令牌而不影响性能，压缩率达2.3倍，仍优于以往启发式自适应方法。通过按信息丰富度分配令牌，InfoTok实现了更紧凑且准确的视频表示，为未来研究提供重要启示。
原文摘要 · Abstract (English)
Accurate and efficient discrete video tokenization is essential for long video sequences processing. Yet, the inherent complexity and variable information density of videos present a significant bottleneck for current tokenizers, which rigidly compress all content at a fixed rate, leading to redundancy or information loss. Drawing inspiration from Shannon's information theory, this paper introduces InfoTok, a principled framework for adaptive video tokenization. We rigorously prove that existing data-agnostic training methods are suboptimal in representation length, and present a novel evidence lower bound (ELBO)-based algorithm that approaches theoretical optimality. Leveraging this framework, we develop a transformer-based adaptive compressor that enables adaptive tokenization. Empirical results demonstrate state-of-the-art compression performance, saving 20% tokens without influence on performance, and achieving 2.3x compression rates while still outperforming prior heuristic adaptive approaches. By allocating tokens according to informational richness, InfoTok enables a more compressed yet accurate tokenization for video representation, offering valuable insights for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。