FlexTok让图像生成可自适应长度的1D令牌序列,提升效率与质量。
FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
- 将图像转为可变长度的有序1D令牌序列,按复杂度动态调整
- 在ImageNet上8-128个令牌即达FID<2,优于TiTok且用更少令牌
- 适合需要灵活生成粒度的任务,如从粗到细的视觉描述
图像分词已通过提供压缩的离散表示推动自回归图像生成的发展。传统方法使用2D网格分词,而近期如TiTok等方法证明1D分词可消除网格冗余并实现高质量生成。但这些方法通常采用固定数量的令牌,无法适应图像内在复杂度。我们提出FlexTok,一种将2D图像投影为可变长度、有序1D令牌序列的分词器。例如,256x256图像可被重采样为1至256个离散令牌,实现层次化语义压缩。通过训练修正流模型作为解码器并使用嵌套丢弃,无论选择何种令牌长度,都能生成合理重建。我们在简单的GPT风格Transformer中评估该方法,在ImageNet上,8至128个令牌即可实现FID<2,优于TiTok,并达到顶尖方法水平,但所用令牌远少于后者。我们进一步扩展模型以支持文本条件图像生成,并分析FlexTok与传统2D分词的关系。关键发现是:FlexTok使下一令牌预测能以由粗到精的‘视觉词汇’描述图像,生成所需令牌数取决于任务复杂度。
原文摘要 · Abstract (English)
Image tokenization has enabled major advances in autoregressive image generation by providing compressed, discrete representations that are more efficient to process than raw pixels. While traditional approaches use 2D grid tokenization, recent methods like TiTok have shown that 1D tokenization can achieve high generation quality by eliminating grid redundancies. However, these methods typically use a fixed number of tokens and thus cannot adapt to an image's inherent complexity. We introduce FlexTok, a tokenizer that projects 2D images into variable-length, ordered 1D token sequences. For example, a 256x256 image can be resampled into anywhere from 1 to 256 discrete tokens, hierarchically and semantically compressing its information. By training a rectified flow model as the decoder and using nested dropout, FlexTok produces plausible reconstructions regardless of the chosen token sequence length. We evaluate our approach in an autoregressive generation setting using a simple GPT-style Transformer. On ImageNet, this approach achieves an FID<2 across 8 to 128 tokens, outperforming TiTok and matching state-of-the-art methods with far fewer tokens. We further extend the model to support to text-conditioned image generation and examine how FlexTok relates to traditional 2D tokenization. A key finding is that FlexTok enables next-token prediction to describe images in a coarse-to-fine "visual vocabulary", and that the number of tokens to generate depends on the complexity of the generation task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。