用分层编码器让文本生成视频更高效，压缩率提升70%
HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models
- 分层离散码本编码视频，高层语义压缩，底层保留细节
- 8秒视频（64帧）比特率降低约70%，重建质量仍优秀
- 适合追求高效生成与模型简化的视频生成研究者
文本到视频生成因视频数据在时空维度上的复杂性而面临挑战，引入了额外冗余、突变以及语言与视觉令牌间的领域差距。为此需要高效的视频分词器，在保持关键语义和时空信息的同时有效编码视频数据，作为文本与视觉之间的关键桥梁。受VQ-VAE-2及传统动画流程启发，本文提出HiTVideo，采用3D因果变分自编码器与多层离散令牌框架，将视频内容编码为层次化结构的码本。高层捕捉语义信息并实现更高压缩，低层聚焦精细时空细节，平衡压缩效率与重建质量。该方法可高效编码更长视频序列（如8秒，64帧），相比基线分词器比特数每像素（bpp）降低约70%，同时保持良好重建质量。我们探讨了压缩与重建间的权衡，强调高压缩语义令牌在文本到视频任务中的优势。HiTVideo旨在解决现有分词器在文本到视频生成中的潜在局限，推动更高压缩比与更简洁的语言模型建模，为文本到视频生成提供可扩展且有前景的框架。演示页：https://ziqinzhou66.github.io/project/HiTVideo。
原文摘要 · Abstract (English)
Text-to-video generation poses significant challenges due to the inherent complexity of video data, which spans both temporal and spatial dimensions. It introduces additional redundancy, abrupt variations, and a domain gap between language and vision tokens while generation. Addressing these challenges requires an effective video tokenizer that can efficiently encode video data while preserving essential semantic and spatiotemporal information, serving as a critical bridge between text and vision. Inspired by the observation in VQ-VAE-2 and workflows of traditional animation, we propose HiTVideo for text-to-video generation with hierarchical tokenizers. It utilizes a 3D causal VAE with a multi-layer discrete token framework, encoding video content into hierarchically structured codebooks. Higher layers capture semantic information with higher compression, while lower layers focus on fine-grained spatiotemporal details, striking a balance between compression efficiency and reconstruction quality. Our approach efficiently encodes longer video sequences (e.g., 8 seconds, 64 frames), reducing bits per pixel (bpp) by approximately 70\% compared to baseline tokenizers, while maintaining competitive reconstruction quality. We explore the trade-offs between compression and reconstruction, while emphasizing the advantages of high-compressed semantic tokens in text-to-video tasks. HiTVideo aims to address the potential limitations of existing video tokenizers in text-to-video generation tasks, striving for higher compression ratios and simplify LLMs modeling under language guidance, offering a scalable and promising framework for advancing text to video generation. Demo page: https://ziqinzhou66.github.io/project/HiTVideo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。