根据视频内容动态压缩冗余帧,实现高效视频编码。
Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting

- 通过分析帧间隐空间差异,自动识别并剔除冗余位置。
- 在DAVIS和TokenBench上实现31倍推理加速,重建质量保持领先。
- 无需额外网络,适合追求高效视频处理的研究与工程人员。
自适应视频分词旨在根据序列的视觉复杂度动态分配分词预算。现有连续方法依赖迭代二值化搜索或训练好的神经回归器,离散方法则常需完整率解码器来估计信息量。我们证明这些计算开销并非必需:冻结的连续视频分词器的隐空间天然包含时间冗余信息——连续帧间隐表示变化极小的位置几乎不携带新信息。本文提出一种无参数的自适应分词机制,对每个位置的时间L1差施加固定阈值,识别并丢弃冗余隐空间位置。由此,压缩率自然由输入内容决定:静态场景可大幅压缩,动态序列保留更多分词。为恢复被丢弃位置,我们设计了轻量级的隐空间修复变压器(LIT),采用分解式时空注意力结构。整个推理流程仅需一次编码器前传和一次LIT前传,无需辅助路由网络。在TokenBench与DAVIS基准上的评估表明,本框架实现了有意义的内容驱动分词,保持良好重建保真度,相较连续自适应基线(ElasticTok-CV)实现31倍推理速度提升,较离散信息论基线(InfoTok)提升约2倍。
原文摘要 · Abstract (English)
Adaptive video tokenisation seeks to dynamically allocate token budgets based on the underlying visual complexity of a sequence. Current continuous-regime approaches achieve this via iterative binarised searches or trained neural regressors, while discrete methods often require a full-rate decoder pass to estimate information content. We demonstrate that such computational overheads are not strictly necessary. We show that the latent space of a frozen continuous video tokeniser inherently encodes temporal redundancy that can be exploited directly: spatial positions whose latent representations change minimally between consecutive frames carry near-zero additional information. We introduce a parameter-free adaptive token allocation mechanism that applies a fixed threshold to per-position temporal-L1 differences, identifying and dropping redundant latent positions. Consequently, the compression rate emerges naturally from the input content rather than being enforced top-down: static scenes get compressed aggressively, while highly dynamic sequences retain more tokens. To reconstruct the dropped positions, we propose the Latent Inpainting Transformer (LIT), a lightweight factorised spatial-temporal attention architecture. The resulting inference pipeline is highly efficient, requiring only a single encoder pass and one LIT forward pass, eliminating the need for auxiliary routing networks. Evaluations across TokenBench and DAVIS, which are the standard benchmarks used by recent tokenisers~\cite{infotok, agarwal2025cosmos}, indicate that our framework yields meaningful, content-driven token allocation while maintaining competitive reconstruction fidelity, and delivers a $31\times$ inference-time speedup over the continuous adaptive baseline (ElasticTok-CV) and an $\approx2\times$ speedup over the discrete information-theoretic baseline (InfoTok).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。