arXiv:2412.01762cs.CV2024-12被引 18

XQ-GAN开源图像分块框架,提升生成与重建质量

XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generation

  • 集成多种先进量化技术,灵活可配置训练环境
  • ImageNet上rFID达0.64,显著优于MAGVIT-v2和VAR
  • 适合作为生成模型的分块器,助力高质量图像生成

图像分块器对后续生成模型性能至关重要。自VQ-GAN问世以来,离散图像分块技术取得显著进展,架构优化、量化方法改进及训练策略完善极大提升了图像重建与生成质量。本文提出XQ-GAN,一个面向图像重建与生成任务的图像分块框架。该框架整合了向量量化(VQ)、残差量化(RQ)、多尺度残差量化(MSVQ)、产品量化(PQ)、无查表量化(LFQ)及二值球面量化(BSQ)等前沿量化技术,构建在高度灵活可定制的训练环境中。在标准ImageNet 256x256基准上,其发布模型rFID达0.64,显著优于MAGVIT-v2(0.9 rFID)和VAR(0.9 rFID)。此外,使用XQ-GAN作为分块器能同时提升gFID指标:相同VAR架构下,XQ-GAN+VAR的gFID为2.6,优于VAR的3.3。为支持后续研究,我们公开了多种预训练分块器权重,供社区直接用于训练生成模型或微调特定任务。

原文摘要 · Abstract (English)

Image tokenizers play a critical role in shaping the performance of subsequent generative models. Since the introduction of VQ-GAN, discrete image tokenization has undergone remarkable advancements. Improvements in architecture, quantization techniques, and training recipes have significantly enhanced both image reconstruction and the downstream generation quality. In this paper, we present XQ-GAN, an image tokenization framework designed for both image reconstruction and generation tasks. Our framework integrates state-of-the-art quantization techniques, including vector quantization (VQ), residual quantization (RQ), multi-scale residual quantization (MSVQ), product quantization (PQ), lookup-free quantization (LFQ), and binary spherical quantization (BSQ), within a highly flexible and customizable training environment. On the standard ImageNet 256x256 benchmark, our released model achieves an rFID of 0.64, significantly surpassing MAGVIT-v2 (0.9 rFID) and VAR (0.9 rFID). Furthermore, we demonstrate that using XQ-GAN as a tokenizer improves gFID metrics alongside rFID. For instance, with the same VAR architecture, XQ-GAN+VAR achieves a gFID of 2.6, outperforming VAR's 3.3 gFID by a notable margin. To support further research, we provide pre-trained weights of different image tokenizers for the community to directly train the subsequent generative models on it or fine-tune for specialized tasks.

图像生成分块器VQ-GAN开源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。