用动态模型大小提升生成效率，3倍少算力仍保持好效果
Masked Generative Nested Transformers with Decode Time Scaling
- 生成时按需调整模型规模，关键步骤用大模型，简单步骤用小模型
- 在ImageNet、UCF101等数据集上，算力减少3倍，性能接近基线
- 适合追求高效率图像/视频生成的开发者或部署场景
视觉生成近年取得显著进展，但推理效率普遍受限。多数方法需多次遍历Transformer模型生成或去噪，且每轮使用相同规模模型，计算开销大。本文提出两个核心思路：(a) 生成过程各阶段计算需求不均，设计解码时模型规模调度策略以优化算力分配；(b) 可缓存并复用部分计算结果。结合二者，让小模型处理更多令牌，大模型处理较少令牌，且模型参数共享，不增加总参数量。在ImageNet256×256、UCF101和Kinetics600数据集上的实验表明，相比基线方法，本方法仅需约1/3计算量即可获得相当性能。
原文摘要 · Abstract (English)
Recent advances in visual generation have made significant strides in producing content of exceptional quality. However, most methods suffer from a fundamental problem - a bottleneck of inference computational efficiency. Most of these algorithms involve multiple passes over a transformer model to generate tokens or denoise inputs. However, the model size is kept consistent throughout all iterations, which makes it computationally expensive. In this work, we aim to address this issue primarily through two key ideas - (a) not all parts of the generation process need equal compute, and we design a decode time model scaling schedule to utilize compute effectively, and (b) we can cache and reuse some of the computation. Combining these two ideas leads to using smaller models to process more tokens while large models process fewer tokens. These different-sized models do not increase the parameter size, as they share parameters. We rigorously experiment with ImageNet256$\times$256 , UCF101, and Kinetics600 to showcase the efficacy of the proposed method for image/video generation and frame prediction. Our experiments show that with almost $3\times$ less compute than baseline, our model obtains competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。