用预训练VAE快速生成高效视觉编码器，训练时间缩至一天内。
Quantize-then-Rectify: Efficient VQ-VAE Training
- 先量化再校正：利用预训练VAE，通过分组量化和后处理修复误差
- 单卡4090仅需22小时完成训练，比顶尖方法快100倍以上
- 压缩图像至512个离散令牌，重建质量媲美先进模型
视觉分词器在多模态大模型中起关键作用，连接连续输入与离散标记。然而，高压缩率的VQ-VAE训练仍计算开销巨大，常需数千小时GPU时间。本文提出 extbf{Quantize-then-Rectify (ReVQ)}框架，通过控制预训练VAE中的量化噪声在容忍阈值内，实现高效VQ-VAE训练。引入 extbf{通道多组量化}以扩大码本容量，并设计 extbf{后校正器}缓解量化误差。ReVQ可将ImageNet图像压缩至最多512个令牌，同时保持优异重建质量（rFID = 1.06）。显著的是，相比现有方法，其训练成本降低两个数量级以上：在单张NVIDIA 4090上约22小时完成全量训练，而同类方法需32块A100 GPU运行4.5天。实验表明，ReVQ在效率与重建质量之间取得更优权衡。
原文摘要 · Abstract (English)
Visual tokenizers are pivotal in multimodal large models, acting as bridges between continuous inputs and discrete tokens. Nevertheless, training high-compression-rate VQ-VAEs remains computationally demanding, often necessitating thousands of GPU hours. This work demonstrates that a pre-trained VAE can be efficiently transformed into a VQ-VAE by controlling quantization noise within the VAE's tolerance threshold. We present \textbf{Quantize-then-Rectify (ReVQ)}, a framework leveraging pre-trained VAEs to enable rapid VQ-VAE training with minimal computational overhead. By integrating \textbf{channel multi-group quantization} to enlarge codebook capacity and a \textbf{post rectifier} to mitigate quantization errors, ReVQ compresses ImageNet images into at most 512 tokens while sustaining competitive reconstruction quality (rFID = 1.06). Significantly, ReVQ reduces training costs by over two orders of magnitude relative to state-of-the-art approaches: ReVQ finishes full training on a single NVIDIA 4090 in approximately 22 hours, whereas comparable methods require 4.5 days on 32 A100 GPUs. Experimental results show that ReVQ achieves superior efficiency-reconstruction trade-offs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。