让图像生成更高效:通过冗余填充和分层语义对齐,激活尾部令牌
Improving Flexible Image Tokenizers for Autoregressive Image Generation
- 引入冗余令牌填充,提升尾部令牌使用频率
- 在ImageNet上生成质量优于现有灵活与固定长度分词器
- 适合追求高质量自回归图像生成的研究者
灵活图像分词器通过嵌套丢弃将图像表示为可变长度的一维令牌序列。传统方法采用尾部截断策略,训练时随机丢弃后段令牌,导致信息过度集中于前段令牌,限制了长序列下自回归图像生成的效果。为此,本文提出ReToK,一种具有冗余令牌填充和分层语义正则化的灵活分词器,以充分挖掘所有令牌的潜在建模能力。具体地,冗余令牌填充使尾部令牌更频繁被激活,缓解信息集中问题;同时,分层语义正则化将早期令牌解码特征与预训练视觉基础模型对齐,并逐步减弱尾部正则强度,以支持低级细节重建。大量实验表明,ReToK在ImageNet 256×256数据集上显著优于现有灵活与固定长度分词器。
原文摘要 · Abstract (English)
Flexible image tokenizers aim to represent an image using an ordered 1D variable-length token sequence. This flexible tokenization is typically achieved through nested dropout, where a portion of trailing tokens is randomly truncated during training, and the image is reconstructed using the remaining preceding sequence. However, this tail-truncation strategy inherently concentrates the image information in the early tokens, limiting the effectiveness of downstream AutoRegressive (AR) image generation as the token length increases. To overcome these limitations, we propose \textbf{ReToK}, a flexible tokenizer with \underline{Re}dundant \underline{Tok}en Padding and Hierarchical Semantic Regularization, designed to fully exploit all tokens for enhanced latent modeling. Specifically, we introduce \textbf{Redundant Token Padding} to activate tail tokens more frequently, thereby alleviating information over-concentration in the early tokens. In addition, we apply \textbf{Hierarchical Semantic Regularization} to align the decoding features of earlier tokens with those from a pre-trained vision foundation model, while progressively reducing the regularization strength toward the tail to allow finer low-level detail reconstruction. Extensive experiments demonstrate the effectiveness of ReTok: on ImageNet 256$\times$256, our method achieves superior generation performance compared with both flexible and fixed-length tokenizers. Code will be available at: \href{https://github.com/zfu006/ReTok}{https://github.com/zfu006/ReTok}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。