用集合自回归模型突破传统图像生成顺序限制，提升生成效率与灵活性。
Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling
- 将图像生成从固定顺序转为任意分组输出，实现更灵活的自回归建模。
- 在ImageNet上验证不同顺序和输出间隔对性能的影响，支持多分辨率生成。
- 可无缝切换自回归与掩码自回归，兼顾推理速度与缓存加速优势，适合高效生成场景。
我们提出一种新的自回归图像生成范式——集合自回归建模（SAR），将传统自回归扩展到任意分组的输出设置，不再局限于固定的光栅扫描顺序。为此，我们设计了全掩码Transformer架构。研究表明，现有自回归变体可视为SAR框架下特定序列顺序与输出间隔的选择，自回归（AR）与掩码自回归（MAR）为其两个极端实例。特别地，SAR实现了从AR到MAR的平滑过渡，中间状态允许训练出兼具少步推理与键值缓存加速优势的因果模型。我们在ImageNet上系统分析了序列顺序与输出间隔对性能及推理顺序/步数泛化能力的影响。进一步验证中，我们训练了一个900M参数的文本到图像模型，可生成任意分辨率的照片级真实图像。本工作希望激发更多基于自回归建模在多模态任务中的探索与应用。
原文摘要 · Abstract (English)
We introduce a new paradigm for AutoRegressive (AR) image generation, termed Set AutoRegressive Modeling (SAR). SAR generalizes the conventional AR to the next-set setting, i.e., splitting the sequence into arbitrary sets containing multiple tokens, rather than outputting each token in a fixed raster order. To accommodate SAR, we develop a straightforward architecture termed Fully Masked Transformer. We reveal that existing AR variants correspond to specific design choices of sequence order and output intervals within the SAR framework, with AR and Masked AR (MAR) as two extreme instances. Notably, SAR facilitates a seamless transition from AR to MAR, where intermediate states allow for training a causal model that benefits from both few-step inference and KV cache acceleration, thus leveraging the advantages of both AR and MAR. On the ImageNet benchmark, we carefully explore the properties of SAR by analyzing the impact of sequence order and output intervals on performance, as well as the generalization ability regarding inference order and steps. We further validate the potential of SAR by training a 900M text-to-image model capable of synthesizing photo-realistic images with any resolution. We hope our work may inspire more exploration and application of AR-based modeling across diverse modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。