arXiv:2509.10140cs.CV2025-09被引 9

提出VQBridge方法,实现向量量化网络100%码本利用率。

Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization

  • 设计压缩-处理-恢复管道,通过映射函数稳定训练码本。
  • 在26.2万码本下仍保持100%利用率,重建性能达顶尖水平。
  • 适用于多种模型架构,显著提升图像生成质量。

向量量化(VQ)是图像生成中离散分词器的核心组件,但其训练常因直通估计偏差、单步滞后更新和稀疏码本梯度而不稳定,导致重建性能不佳与码本利用率低。本文分析上述根本挑战并提出VQBridge——一种基于映射函数的鲁棒、可扩展且高效的投影器。VQBridge通过压缩-处理-恢复流程优化码向量,实现稳定训练。结合学习率退火,所提VQN在多种码本配置下均实现100%码本利用率,称为FVQ(FullVQ)。大量实验表明,FVQ具有优异的可扩展性与泛化能力:即使在26.2万码本规模下仍保持100%利用率,重建性能达到当前最优,且随码本增大、通道数增加或训练时长延长持续提升,并适用于不同VQ变体。当集成至LlamaGen时,FVQ使图像生成性能超越视觉自回归模型(VAR)0.5 rFID,扩散模型(DiT)0.2 rFID,凸显高质量分词器对强大自回归图像生成的重要性。

原文摘要 · Abstract (English)

Vector quantization (VQ) is a key component in discrete tokenizers for image generation, but its training is often unstable due to straight-through estimation bias, one-step-behind updates, and sparse codebook gradients, which lead to suboptimal reconstruction performance and low codebook usage. In this work, we analyze these fundamental challenges and provide a simple yet effective solution. To maintain high codebook usage in VQ networks (VQN) during learning annealing and codebook size expansion, we propose VQBridge, a robust, scalable, and efficient projector based on the map function method. VQBridge optimizes code vectors through a compress-process-recover pipeline, enabling stable and effective codebook training. By combining VQBridge with learning annealing, our VQN achieves full (100%) codebook usage across diverse codebook configurations, which we refer to as FVQ (FullVQ). Through extensive experiments, we demonstrate that FVQ is effective, scalable, and generalizable: it attains 100% codebook usage even with a 262k-codebook, achieves state-of-the-art reconstruction performance, consistently improves with larger codebooks, higher vector channels, or longer training, and remains effective across different VQ variants. Moreover, when integrated with LlamaGen, FVQ significantly enhances image generation performance, surpassing visual autoregressive models (VAR) by 0.5 and diffusion models (DiT) by 0.2 rFID, highlighting the importance of high-quality tokenizers for strong autoregressive image generation.

向量量化图像生成码本利用训练稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。