用大小模型协作解码，让图像自回归生成更快更省显存
Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
- 大模型负责低频内容生成，小模型专注高频细节，分工协同
- 在256×256分辨率下实现41张/秒，速度提升2.9倍，显存减半
- 适合追求高效图像生成的开发者，尤其适合资源受限场景
在快速发展的图像生成领域，视觉自回归（VAR）建模因其创新的逐尺度预测方法受到广泛关注。该范式在效率、可扩展性和零样本泛化方面有显著提升。然而，VAR固有的粗到细特性导致长序列生成，带来巨大的内存消耗和计算冗余。为此，我们提出协同解码（CoDe），一种专为VAR框架设计的高效解码策略。CoDe基于两个关键观察：大规模下参数需求显著降低，以及不同尺度间生成模式具有独占性。据此，将多尺度推理过程划分为大模型与小模型的无缝协作：大模型作为‘起草者’，专注于小尺度的低频内容生成；小模型作为‘精修者’，仅负责大尺度的高频细节预测。该协作实现显著效率提升且质量损失极小：速度提升1.7倍，内存减少约50%，图像质量仅轻微下降（FID从1.95升至1.98）。当减少起草步骤时，加速比可达2.9倍，在单张NVIDIA 4090 GPU上实现41张/秒的生成速度，同时保持2.27的优良FID。代码已开源。
原文摘要 · Abstract (English)
In the rapidly advancing field of image generation, Visual Auto-Regressive (VAR) modeling has garnered considerable attention for its innovative next-scale prediction approach. This paradigm offers substantial improvements in efficiency, scalability, and zero-shot generalization. Yet, the inherently coarse-to-fine nature of VAR introduces a prolonged token sequence, leading to prohibitive memory consumption and computational redundancies. To address these bottlenecks, we propose Collaborative Decoding (CoDe), a novel efficient decoding strategy tailored for the VAR framework. CoDe capitalizes on two critical observations: the substantially reduced parameter demands at larger scales and the exclusive generation patterns across different scales. Based on these insights, we partition the multi-scale inference process into a seamless collaboration between a large model and a small model. The large model serves as the 'drafter', specializing in generating low-frequency content at smaller scales, while the smaller model serves as the 'refiner', solely focusing on predicting high-frequency details at larger scales. This collaboration yields remarkable efficiency with minimal impact on quality: CoDe achieves a 1.7x speedup, slashes memory usage by around 50%, and preserves image quality with only a negligible FID increase from 1.95 to 1.98. When drafting steps are further decreased, CoDe can achieve an impressive 2.9x acceleration ratio, reaching 41 images/s at 256x256 resolution on a single NVIDIA 4090 GPU, while preserving a commendable FID of 2.27. The code is available at https://github.com/czg1225/CoDe
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。