用小波变换的频谱分块，让图像生成更高效清晰。
Spectral Image Tokenizer
- 将图像转为小波频谱分块，从粗到细生成。
- 支持多分辨率重建，生成质量优于传统方法。
- 适合需要快速预览或图像放大场景。
图像标记器将图像映射为离散标记序列，是基于自回归Transformer的图像生成的关键组件。传统标记器按栅格扫描顺序排列空间位置，不利于自回归建模。本文提出对图像频谱进行标记，通过离散小波变换（DWT）实现，使标记序列以由粗到细的方式表示图像。该方法具有多项优势：1）利用自然图像在高频区域更具可压缩性的特性；2）无需重新训练即可处理不同分辨率的图像；3）改善下一标记预测的条件性——不再依赖逐行部分重建，而是基于全图粗略重建；4）支持部分解码，前几项生成标记即可还原图像粗略版本；5）使自回归模型可用于图像上采样。我们在重建指标、多尺度图像生成、文本引导的图像上采样与编辑任务中进行了评估。
原文摘要 · Abstract (English)
Image tokenizers map images to sequences of discrete tokens, and are a crucial component of autoregressive transformer-based image generation. The tokens are typically associated with spatial locations in the input image, arranged in raster scan order, which is not ideal for autoregressive modeling. In this paper, we propose to tokenize the image spectrum instead, obtained from a discrete wavelet transform (DWT), such that the sequence of tokens represents the image in a coarse-to-fine fashion. Our tokenizer brings several advantages: 1) it leverages that natural images are more compressible at high frequencies, 2) it can take and reconstruct images of different resolutions without retraining, 3) it improves the conditioning for next-token prediction -- instead of conditioning on a partial line-by-line reconstruction of the image, it takes a coarse reconstruction of the full image, 4) it enables partial decoding where the first few generated tokens can reconstruct a coarse version of the image, 5) it enables autoregressive models to be used for image upsampling. We evaluate the tokenizer reconstruction metrics as well as multiscale image generation, text-guided image upsampling and editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。