用分层向量量化实现高速高保真生成，采样速度不随深度增加而下降。
Efficient Generative Modeling with Residual Vector Quantization-Based Tokens
- 直接预测一组向量嵌入，避免逐个生成，保持采样步数不变。
- 在ImageNet和零样本语音合成上超越自回归模型，速度与质量双优。
- 支持扩展量化深度，可提升画质或加快速度，适合多模态生成任务。
我们提出ResGen，一种基于残差向量量化（RVQ）的高效生成模型，可在保证高保真度的同时实现快速采样。传统RVQ通过增加量化层数（深度）提升数据保真度，但通常导致生成过程推理步数增加。ResGen通过直接预测多个量化向量的联合嵌入，使推理步数与RVQ深度无关。此外，我们在概率框架下整合了令牌掩码与多令牌预测，结合离散扩散与变分推断。在两个跨模态挑战任务上验证：ImageNet 256x256上的条件图像生成与零样本文本到语音合成。实验表明，ResGen在两项任务中均优于自回归模型，在不牺牲采样速度的前提下实现更优性能。随着RVQ深度增加，生成模型在相同规模下表现出更高的生成保真度或更快的采样速度，显著优于基线模型。
原文摘要 · Abstract (English)
We introduce ResGen, an efficient Residual Vector Quantization (RVQ)-based generative model for high-fidelity generation with fast sampling. RVQ improves data fidelity by increasing the number of quantization steps, referred to as depth, but deeper quantization typically increases inference steps in generative models. To address this, ResGen directly predicts the vector embedding of collective tokens rather than individual ones, ensuring that inference steps remain independent of RVQ depth. Additionally, we formulate token masking and multi-token prediction within a probabilistic framework using discrete diffusion and variational inference. We validate the efficacy and generalizability of the proposed method on two challenging tasks across different modalities: conditional image generation on ImageNet 256x256 and zero-shot text-to-speech synthesis. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sampling speeds compared to similarly sized baseline models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。