让图像生成的分词器与自回归模型对齐，提升生成质量与速度。
Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model
- 设计双向编码+因果解码的分词器，使序列具备前向依赖性。
- 177M模型在ImageNet-256上达gFID 1.44，662M模型更快更优。
- 适合关注高效高质图像生成的研究者与开发者。
自回归图像生成需基于前序标记预测下一标记，但传统图像分词方式存在双向依赖，与自回归模型单向性不匹配。为此，本文提出AliTok，一种新型对齐分词器，通过双向编码器配合因果解码器约束，使编码器输出的标记序列兼具语义丰富性与前向依赖性。结合前缀标记和两阶段训练策略，显著提升重建性能。基于AliTok，仅177M参数的解码器模型在ImageNet-256上实现gFID 1.44、IS 319.5；扩展至662M参数时，gFID降至1.28，采样速度比现有扩散方法快10倍；在ImageNet-512上，318M模型亦达SOTA gFID 1.39。代码与权重见https://github.com/ali-vilab/alitok。
原文摘要 · Abstract (English)
Autoregressive image generation aims to predict the next token based on previous ones. However, this process is challenged by the bidirectional dependencies inherent in conventional image tokenizations, which creates a fundamental misalignment with the unidirectional nature of autoregressive models. To resolve this, we introduce AliTok, a novel Aligned Tokenizer that alters the dependency structure of the token sequence. AliTok employs a bidirectional encoder constrained by a causal decoder, a design that compels the encoder to produce a token sequence with both semantic richness and forward-dependency. Furthermore, by incorporating prefix tokens and employing a two-stage tokenizer training process to enhance reconstruction performance, AliTok achieves high fidelity and predictability simultaneously. Building upon AliTok, a standard decoder-only autoregressive model with just 177M parameters achieves a gFID of 1.44 and an IS of 319.5 on ImageNet-256. Scaling to 662M, our model reaches a gFID of 1.28, surpassing the SOTA diffusion method with 10x faster sampling. On ImageNet-512, our 318M model also achieves a SOTA gFID of 1.39. Code and weights at https://github.com/ali-vilab/alitok.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。